Lip-tracking hearing aid

ABSTRACT

A system may include a wearable camera configured to capture a plurality of images from an environment of a user and a microphone configured to capture sounds from an environment of the user. The system may also include a processor programmed to receive the images; identify a representation of one individual in one of the images; identify a lip movement associated with a mouth of the individual, based on analysis of the images; receive audio signals representative of the sounds; identify, based on analysis of the sounds, a first audio signal associated with a first voice and a second audio signal associated with a second voice; cause selective conditioning of the first audio signal based on a determination that the first audio signal is associated with the identified lip movement; and cause transmission of the selectively conditioned first audio signal to a hearing interface device.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/888,588, filed May 29, 2020, which is a continuation of InternationalApplication No. PCT/IB2019/001132, filed Oct. 10, 2019, which claims thebenefit of priority of U.S. Provisional Patent Application No.62/745,478, filed on Oct. 15, 2018; U.S. Provisional Patent ApplicationNo. 62/746,595, filed on Oct. 17, 2018; U.S. Provisional PatentApplication No. 62/808,317, filed on Feb. 21, 2019; and U.S. ProvisionalPatent Application No. 62/857,773, filed on Jun. 5, 2019. All of theforegoing applications are incorporated herein by reference in theirentirety.

BACKGROUND Technical Field

This disclosure generally relates to devices and methods for capturingand processing images and audio from an environment of a user, and usinginformation derived from captured images and audio.

Background Information

Today, technological advancements make it possible for wearable devicesto automatically capture images and audio, and store information that isassociated with the captured images and audio. Certain devices have beenused to digitally record aspects and personal experiences of one's lifein an exercise typically called “lifelogging.” Some individuals logtheir life so they can retrieve moments from past activities, forexample, social events, trips, etc. Lifelogging may also havesignificant benefits in other fields (e.g., business, fitness andhealthcare, and social research). Lifelogging devices, while useful fortracking daily activities, may be improved with capability to enhanceone's interaction in his environment with feedback and other advancedfunctionality based on the analysis of captured image and audio data.

Even though users can capture images and audio with their smartphonesand some smartphone applications can process the captured information,smartphones may not be the best platform for serving as lifeloggingapparatuses in view of their size and design. Lifelogging apparatusesshould be small and light, so they can be easily worn. Moreover, withimprovements in image capture devices, including wearable apparatuses,additional functionality may be provided to assist users in navigatingin and around an environment, identifying persons and objects theyencounter, and providing feedback to the users about their surroundingsand activities. Therefore, there is a need for apparatuses and methodsfor automatically capturing and processing images and audio to provideuseful information to users of the apparatuses, and for systems andmethods to process and leverage information gathered by the apparatuses.

SUMMARY

Embodiments consistent with the present disclosure provide devices andmethods for automatically capturing and processing images and audio froman environment of a user, and systems and methods for processinginformation related to images and audio captured from the environment ofthe user.

In an embodiment, a hearing aid system may selectively amplify soundsemanating from a detected look direction of a user of the hearing aidsystem. The system may include a wearable camera configured to capture aplurality of images from an environment of the user; at least onemicrophone configured to capture sounds from an environment of the user;and at least one processor. The processor may be programmed to receivethe plurality of images captured by the camera, receive audio signalsrepresentative of sounds received by the at least one microphone fromthe environment of the user, determine a look direction for the userbased on analysis of at least one of the plurality of images, causeselective conditioning of at least one audio signal received by the atleast one microphone from a region associated with the look direction ofthe user, and cause transmission of the at least one conditioned audiosignal to a hearing interface device configured to provide sound to anear of the user.

In an embodiment, a method may selectively amplify sounds emanating froma detected look direction of a user of the hearing aid system. Themethod may comprise receiving a plurality of images captured by awearable camera from an environment of a user; receiving audio signalsrepresentative of sounds captured by at least one microphone from theenvironment of the user, determining a look direction for the user basedon analysis of at least one of the plurality of images, causingselective conditioning of at least one audio signal received by the atleast one microphone from a region associated with the look direction ofthe user, causing transmission of the at least one conditioned audiosignal to a hearing interface device configured to provide sound to anear of the user.

In an embodiment, a hearing aid system may selectively amplify audiosignals associated with a voice of a recognized individual. The systemmay include a wearable camera configured to capture a plurality ofimages from an environment of the user, at least one microphoneconfigured to capture sounds from an environment of the user, and atleast one processor. The processor may be programmed to receive theplurality of images captured by the camera, identify a representation ofat least one recognized individual in at least one of the plurality ofimages, receive audio signals representative of the sounds captured bythe at least one microphone, cause selective conditioning of at leastone audio signal received by the at least one microphone from a regionassociated with the at least one recognized individual, and causetransmission of the at least one conditioned audio signal to a hearinginterface device configured to provide sound to an ear of the user.

In an embodiment, a method may selectively amplify audio signalsassociated with a voice of a recognized individual. The method maycomprise receiving a plurality of images captured by a wearable camerafrom an environment of the user, identifying a representation of atleast one recognized individual in at least one of the plurality ofimages, receiving audio signals representative of sounds captured by atleast one microphone from the environment of the user, causing selectiveconditioning of at least one audio signal received by the at least onemicrophone from a region associated with the at least one recognizedindividual, and causing transmission of the at least one conditionedaudio signal to a hearing interface device configured to provide soundto an ear of the user.

In an embodiment, a voice transmission system may selectively transmitaudio signals associated with a voice of a recognized user. The systemmay include at least one microphone configured to capture sounds from anenvironment of the user and at least one processor. The processor may beprogrammed to receive audio signals representative of the soundscaptured by the at least one microphone, identify, based on analysis ofthe received audio signals, one or more voice audio signalsrepresentative of a recognized voice of the user, cause transmission, toa remotely located device, of the one or more voice audio signalsrepresentative of the recognized voice of the user, and preventtransmission, to the remotely located device, of at least one backgroundnoise audio signal different from the one or more voice audio signalsrepresentative of a recognized voice of the user.

In an embodiment, a method may selectively transmit audio signalsassociated with a voice of a recognized user. The method may comprisereceiving audio signals representative of sounds captured by at leastone microphone from an environment of a user, identifying, based onanalysis of the received audio signals, one or more voice audio signalsrepresentative of a recognized voice of the user, causing transmission,to a remotely located device, of the one or more voice audio signalsrepresentative of the recognized voice of the user, and preventingtransmission, to the remotely located device, of at least one backgroundnoise audio signal different from the one or more voice audio signalsrepresentative of a recognized voice of the user.

In an embodiment, a hearing aid system may selectively amplify audiosignals based on tracked lip movements. The system may include awearable camera configured to capture a plurality of images from anenvironment of the user, at least one microphone configured to capturesounds from an environment of the user, and at least one processor. Theprocessor may be programmed to receive the plurality of images capturedby the camera; identify a representation of at least one individual inat least one of the plurality of images; identify at least one lipmovement associated with a mouth of the individual, based on analysis ofthe plurality of images; receive audio signals representative of thesounds captured by the at least one microphone; identify, based onanalysis of the sounds captured by the at least one microphone, at leasta first audio signal associated with a first voice and at least a secondaudio signal associated with a second voice different from the firstvoice; cause selective conditioning of the first audio signal based on adetermination by the at least one processor that the first audio signalis associated with the identified at least one lip movement associatedwith the mouth of the individual; and cause transmission of theselectively conditioned first audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user.

In an embodiment, a method may selectively amplify audio signals basedon tracked lip movements. The method may comprise receiving a pluralityof images captured by a wearable camera from an environment of the user;identifying a representation of at least one individual in at least oneof the plurality of images; identifying at least one lip movementassociated with a mouth of the individual, based on analysis of theplurality of images; receiving audio signals representative of thesounds captured by at least one microphone from the environment of theuser; identifying, based on analysis of the sounds captured by the atleast one microphone, at least a first audio signal associated with afirst voice and at least a second audio signal associated with a secondvoice different from the first voice; causing selective conditioning ofthe first audio signal based on a determination by the at least oneprocessor that the first audio signal is associated with the identifiedat least one lip movement associated with the mouth of the individual;and causing transmission of the selectively conditioned first audiosignal to a hearing interface device configured to provide sound to anear of the user.

In an embodiment, a hearing aid system for amplifying audio signals maycomprise a wearable camera configured to capture a plurality of imagesfrom an environment of a user and at least one microphone configured tocapture sounds from an environment of the user. The hearing aid systemmay also include at least one processor programmed to receive theplurality of images captured by the camera and identify a representationof a first individual and a representation of a second individual in theplurality of images. The at least one processor may also be programmedto receive from the at least one microphone a first audio signalassociated with a voice of the first individual and receive from the atleast one microphone a second audio signal associated with a voice ofthe second individual. The at least one processor may further beprogrammed to detect at least one amplification criteria indicative of avoice amplification priority between the first individual and the secondindividual. The at least one processor may also be programmed toselectively amplify the first audio signal relative to the second audiosignal when the at least one amplification criteria indicates that thefirst individual has voice amplification priority over the secondindividual and selectively amplify the second audio signal relative tothe first audio signal when the at least one amplification criteriaindicates that the second individual has voice amplification priorityover the first individual. The at least one processor may further beprogrammed to cause transmission of the selectively amplified first orsecond audio signal to a hearing interface device configured to providesound to an ear of the user.

In an embodiment, a computer-implemented method for selectivelyamplifying audio signals may comprise receiving the plurality of imagescaptured by a camera from an environment of a user and identifying arepresentation of a first individual and a representation of a secondindividual in the plurality of images. The method may also comprisereceiving from at least one microphone a first audio signal associatedwith a voice of the first individual and receiving from the at least onemicrophone a second audio signal associated with a voice of the secondindividual. The method may further comprise detecting at least oneamplification criteria indicative of a voice amplification prioritybetween the first individual and the second individual. The method mayalso comprise selectively amplifying the first audio signal relative tothe second audio signal when the at least one amplification criteriaindicates that the first individual has voice amplification priorityover the second individual and selectively amplifying the second audiosignal relative to the first audio signal when the at least oneamplification criteria indicates that the second individual has voiceamplification priority over the first individual. The method may furthercomprise causing transmission of the selectively amplified first orsecond audio signal to a hearing interface device configured to providesound to an ear of the user.

In an embodiment, a non-transitory computer-readable medium storeinstructions that, when executed by at least one processor, may cause adevice to perform a method comprising receiving the plurality of imagescaptured by a camera from an environment of a user and identifying arepresentation of a first individual and a representation of a secondindividual in the plurality of images. The method may also comprisereceiving from at least one microphone a first audio signal associatedwith a voice of the first individual and receiving from the at least onemicrophone a second audio signal associated with a voice of the secondindividual. The method may further comprise detecting at least oneamplification criteria indicative of a voice amplification prioritybetween the first individual and the second individual. The method mayalso comprise selectively amplifying the first audio signal relative tothe second audio signal when the at least one amplification criteriaindicates that the first individual has voice amplification priorityover the second individual and selectively amplifying the second audiosignal relative to the first audio signal when the at least oneamplification criteria indicates that the second individual has voiceamplification priority over the first individual. The method may furthercomprise causing transmission of the selectively amplified first orsecond audio signal to a hearing interface device configured to providesound to an ear of the user.

In an embodiment, a hearing aid system for selectively amplifying audiosignals may comprise a wearable camera configured to capture a pluralityof images from an environment of a user and at least one microphoneconfigured to capture sounds from an environment of the user. Thehearing aid system may also include at least one processor programmed toreceive the plurality of images captured by the camera; identify arepresentation of one or more individuals in the plurality of images;receive from the at least one microphone a first audio signal associatedwith a voice; determine, based on analysis of the plurality of images,that the first audio signal is not associated with a voice of any of theone or more individuals; receive from the at least one microphone asecond audio signal associated with a voice; determine, based onanalysis of the plurality of images, that the second audio signal isassociated with a voice of one of the one or more individuals; cause afirst amplification of the first audio signal and a second amplificationof the second audio signal, wherein the first amplification differs fromthe second amplification in at least one aspect; and cause transmissionof at least one of the first audio signal, amplified according to thefirst amplification, and the second audio signal, amplified according tothe second amplification, to a hearing interface device configured toprovide sound to an ear of the user.

In an embodiment, a hearing aid system for selectively amplifying audiosignals may comprise a wearable camera configured to capture a pluralityof images from an environment of a user and at least one microphoneconfigured to capture sounds from an environment of the user. Thehearing aid system may also include at least one processor programmedto: receive a first plurality of images captured by the camera; identifya representation of an individual in the first plurality of images;receive from the at least one microphone a first audio signalrepresentative of a voice; determine, based on analysis of the firstplurality of images, that the first audio signal representative of avoice is associated with the individual; selectively amplify the firstaudio signal over other audio signals received from the at least onemicrophone representative of sounds from sources other than theindividual; receive a second plurality of images captured by the camera;determine, based on analysis of the second plurality of images, that theindividual is not represented in the second plurality of images; receivefrom the at least one microphone a second audio signal representative ofa voice; determine, based on analysis of the first audio signal and thesecond audio signal, that the second audio signal is associated with theindividual; selectively amplify the second audio signal over otherreceived audio signals representative of sounds from sources other thanthe individual; and cause transmission of at least one of theselectively amplified first audio signal or the selectively amplifiedsecond audio signal to a hearing interface device configured to providesound to an ear of the user.

In an embodiment, a hearing aid system for selectively amplifying audiosignals may comprise a wearable camera configured to capture a pluralityof images from an environment of a user and at least one microphoneconfigured to capture sounds from an environment of the user. Thehearing aid system may also include at least one processor programmedto: receive the plurality of images captured by the camera; identify arepresentation of one or more individuals in the plurality of images;receive from the at least one microphone an audio signal associated witha voice; determine, based on analysis of the plurality of images, thatthe audio signal is not associated with a voice of any of the one ormore individuals; determine, based on analysis of the audio signal, thatthe audio signal is associated with at least one indicator that theaudio signal is related to a public announcement; cause selectiveamplification of the audio signal based on the determination that theaudio signal is associated with at least one indicator that the audiosignal relates to a public announcement; and cause transmission of theselectively amplified audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user.

In an embodiment, a hearing aid system is provided. The system mayinclude a wearable camera configured to capture a plurality of imagesfrom an environment of a user; at least one microphone configured tocapture sounds from an environment of the user; and at least oneprocessor. The processor may be programmed to receive the plurality ofimages captured by the camera; identify a representation of at least oneindividual in at least one of the plurality of images, and determinewhether the at least one individual is a recognized individual. Further,if the at least one individual is determined to be a recognizedindividual, cause an image of the at least one individual to be shown ona display and selectively condition at least one audio signal that isreceived from the at least one microphone and determined to beassociated with the recognized individual; and cause transmission of theat least one conditioned audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user.

In an embodiment, a hearing aid system is provided. The system mayinclude a wearable camera configured to capture a plurality of imagesfrom an environment of a user; at least one microphone configured tocapture sounds from an environment of the user; and at least oneprocessor. The processor may be programmed to receive an audio signalfrom the at least one microphone and determine whether the receivedaudio signal is associated with a recognized individual. Further, if theat least one individual is determined to be a recognized individual,cause an image of the at least one individual to be shown on a displayand selectively condition the audio signal, and cause transmission ofthe conditioned audio signal to a hearing interface device configured toprovide sound to an ear of the user.

In an embodiment, a hearing aid system is provided. The system mayinclude a wearable camera configured to capture a plurality of imagesfrom an environment of a user; at least one microphone configured tocapture sounds from an environment of the user; and at least oneprocessor. The processor may be programmed to receive audio signals fromthe at least one microphone; detect, based on analysis of the audiosignals, a first audio signal associated with a first time period,wherein the first audio signal is representative of the voice of asingle individual; detect, based on analysis of the audio signals, asecond audio signal associated with a second time period, wherein thesecond time period is different from the first time period, and whereinthe second audio signal is representative of overlapping voices of twoor more individuals; selectively condition the first audio signal andthe second audio signal, wherein the selective conditioning of the firstaudio signal is different in at least one respect relative the selectiveconditioning of the second audio signal; and cause transmission of theconditioned first audio signal to a hearing interface device configuredto provide sound to an ear of the user.

In an embodiment, a hearing aid system is disclosed. The system includesa wearable camera configured to capture a plurality of images from anenvironment of a user, at least one microphone configured to capturesounds from the environment of the user, and at least one processorprogrammed to: receive the plurality of images captured by the wearablecamera; receive audio signals representative of sounds captured by theat least one microphone; identify a first audio signal, from among thereceived audio signals, representative of a voice of a first individual;transcribe and store, in a memory, text corresponding to speechassociated with the voice of the first individual; determine whether thefirst individual is a recognized individual; and if the first individualis a recognized individual, associate an identifier of the firstrecognized individual with the stored text corresponding to speechassociated with the voice of the first individual.

In an embodiment, a computer-implemented method for individualidentification of a hearing aid system is disclosed. The methodincludes: receiving a plurality of images from a wearable camera;receiving audio signals representative of sounds from at least onemicrophone; identifying a first audio signal, from among the receivedaudio signals, representative of a voice of a first individual;transcribing and storing text corresponding to speech associated withthe voice of the first individual; determining whether the firstindividual is a recognized individual; and if the first individual is arecognized individual, associating an identifier of the first recognizedindividual with the stored text corresponding to speech associated withthe voice of the first individual.

In an embodiment, a non-transitory computer readable storage media isdisclosed. The non-transitory computer readable storage media storesprogram instructions which are executed by at least one processor toperform: receiving a plurality of images from a wearable camera;receiving audio signals representative of sounds from at least onemicrophone; identifying a first individual represented in at least oneof the plurality of images; identifying a first audio signal, from amongthe received audio signals, representative of a voice of the a firstindividual; transcribing and storing text corresponding to speechassociated with the voice of the first individual; determining whetherthe first individual is a recognized individual; and if the firstindividual is a recognized individual, associating an identifier of thefirst recognized individual with the stored text corresponding to thespeech associated with the voice of the first individual.

In an embodiment, a hearing aid system for selectively conditioningaudio signals associated with a recognized object is provided. Thesystem may include at least one processor programmed to receive audiosignals acquired by a wearable microphone, wherein the audio signals arerepresentative of sounds emanating from objects in an environment of auser. The at least one processor may analyze the received audio signalsto obtain an isolated audio stream associated with a sound-emanatingobject in the environment of the user. Further, the at least oneprocessor may determine an audioprint from the isolated audio stream andmay use the audioprint to retrieve from a database information relatingto the particular sound-emanating object. Based on the retrievedinformation, the at least one processor may cause selective conditioningof at least one audio signal received by the wearable microphone from aregion associated with the at least one sound-emanating object, and maycause transmission of the at least one conditioned audio signal to ahearing interface device configured to provide sounds to an ear of theuser.

In an embodiment, a method is provided for selectively conditioningaudio signals associated with a recognized object. The method maycomprise receiving audio signals acquired by a wearable microphone,wherein the audio signals are representative of sounds emanating fromobjects in an environment of a user; analyzing the received audiosignals to isolate an audio stream determined to be associated with aparticular sound-emanating object in the environment of the user;determining an audioprint of the isolated audio stream; using thedetermined audioprint to retrieve from a database information relatingto the particular sound-emanating object; based on the retrievedinformation, causing selective conditioning of at least one audio signalreceived by the wearable microphone from a region associated with the atleast one sound-emanating object; and causing transmission of the atleast one conditioned audio signal to a hearing interface deviceconfigured to provide sounds to an ear of the user.

In an embodiment, a hearing aid system for selectively conditioningaudio signals associated with a recognized object is provided. Thesystem may include at least one processor programmed to receive aplurality of images from an environment of a user captured by a wearablecamera. The at least one processor may process the plurality of imagesto detect a sound-emanating object in at least one of the plurality ofimages, and identify the sound-emanating object using the at least oneof the plurality of images. The at least one processor may further usethe determined identity of the sound-emanating object to retrieve from adatabase information relating to the sound-emanating object. The atleast one processor may also receive at least one audio signal acquiredby a wearable microphone, wherein the at least one audio signal isrepresentative of sounds including a sound emanating from thesound-emanating object, and separate the at least one audio signal usingthe retrieved information to isolate the sound emanating from thesound-emanating object, cause selective conditioning of the sound toobtain at least one conditioned audio signal, and may cause transmissionof the at least one conditioned audio signal to a hearing interfacedevice configured to provide sounds to an ear of the user.

In an embodiment, a hearing aid system for selective modification ofbackground noises is provided. The system may include and at least oneprocessor programmed to receive a plurality of images from anenvironment of a user captured by a wearable camera during a timeperiod, and receive at least one audio signal representative of soundsacquired by a wearable microphone during the time period. Further, theat least one processor may determine that at least one of the sounds wasgenerated by a sound-emanating object in the environment of the user,but outside of a field of view of the wearable camera, and retrieve froma database information associated with the at least one sound. Based onthe retrieved information, the at least one processor may causeselective conditioning of audio signals acquired by the wearablemicrophone during the time period and causes transmission of theconditioned audio signals to a hearing interface device configured toprovide sounds to an ear of the user.

In an embodiment, a method is provided for selective modification ofdifferent types of background noises. The method may comprise receivinga plurality of images from an environment of a user captured by awearable camera during a time period; receiving audio signalsrepresentative of sounds from the environment of the user acquired by awearable microphone during the time period; determining that at leastone of the sounds was generated in response to sounds from asound-emanating object in the environment of the user, but outside of afield of view of the wearable camera; retrieving from a databaseinformation associated with the at least one of the sounds based on theretrieved information, causing selective conditioning of audio signalsacquired by the wearable microphone during the time period; and causingtransmission of the conditioned audio signals to a hearing interfacedevice configured to provide sounds to an ear of the user.

In an embodiment, a system for identifying sound-emanating objects in anenvironment of a user is disclosed. The system may comprise at least onememory device configured to store a database of reference visualcharacteristics and reference voiceprints corresponding to a pluralityof objects; and at least one processor. The processor may be programmedto receive a plurality of images captured by a wearable camera, whereinat least one of the plurality of images depicts at least onesound-emanating object in an environment of a user; analyze the receivedat least one of the plurality of images to determine one or more visualcharacteristics associated with the at least one sound-emanating object;identify within the database in view of the one or more visualcharacteristics, the at least one sound-emanating object and determine adegree of certainty of identification; receive audio signals acquired bya wearable microphone, wherein the audio signals are representative ofone or more sounds emanating from the at least one sound-emanatingobject; analyze the received audio signals to determine a voiceprint ofthe at least one sound-emanating object; when the degree of certainty ofidentification falls below a predetermined level, further identify theat least one sound-emanating object based on the determined voiceprint;and initiate at least one action based on an identity of the at leastone sound-emanating object.

In an embodiment, a method for identifying sound-emanating objects in anenvironment of a user is disclosed. The method may comprise accessing adatabase of reference visual signatures and reference voice signaturescorresponding to a plurality of objects; receiving a plurality of imagescaptured by a wearable camera, wherein at least one of the plurality ofimages depicts at least one sound-emanating object in an environment ofa user; analyzing the received at least one of the plurality of imagesto determine one or more visual characteristics associated with the atleast one sound-emanating object; identifying, based on review of thedatabase in view of the one or more visual characteristics, the at leastone sound-emanating object and determine a degree of certainty ofidentification; receiving audio signals acquired by a wearablemicrophone, wherein the audio signals are representative of one or moresounds emanating from the at least one sound-emanating object; analyzingthe received audio signals to determine voiceprint of the at least onesound-emanating object; when the degree of certainty of identificationfalls below a predetermined level, further identifying the at least onesound-emanating object based on the determined voiceprint; andinitiating at least one action based on an identity of the at least onesound-emanating object.

In an embodiment, a software product may be stored on a non-transitorycomputer readable medium and may comprise computer implementableinstructions for a method for identifying sound-emanating objects. Themethod may comprise accessing a database of reference visual signaturesand reference voice signatures corresponding to a plurality of objects;receiving a plurality of images captured by a wearable camera, whereinat least one of the plurality of images depicts at least onesound-emanating object in an environment of a user; analyzing thereceived at least one of the plurality of images to determine one ormore visual characteristics associated with the at least onesound-emanating object; identifying, based on review of the database inview of the one or more visual characteristics, the at least onesound-emanating object and determine a degree of certainty ofidentification; receiving audio signals acquired by a wearablemicrophone, wherein the audio signals are representative of one or moresounds emanating from the at least one sound-emanating object; analyzingthe received audio signals to determine voiceprint of the at least onesound-emanating object; when the degree of certainty of identificationfalls below a predetermined level, further identifying the at least onesound-emanating object based on the determined voiceprint; andinitiating at least one action based on an identity of the at least onesound-emanating object.

In an embodiment, a hearing aid system may selectively condition audiosignals. The hearing aid system may include at least one processorprogrammed to receive a plurality of images captured by a wearablecamera, wherein the plurality of images depict objects in an environmentof a user; receive audio signals acquired by a wearable microphone,wherein the audio signals are representative of sounds emanating fromthe objects; analyze the plurality of images to identify at least onesound-emanating object in the environment of the user; retrieve from adatabase information about the at least one identified sound-emanatingobject; based on the retrieved information, cause selective conditioningof at least one audio signal received by the wearable microphone from aregion associated with the at least one sound-emanating object; causetransmission of the at least one conditioned audio signal to a hearinginterface device configured to provide sounds to an ear of the user.

In an embodiment, a method for modifying sounds emanating from objectsin an environment of a user is disclosed. The method may comprisereceiving a plurality of images captured by a wearable camera, whereinthe plurality of images depict objects in an environment of a user;receiving audio signals acquired by a wearable microphone, wherein theaudio signals are representative of sounds emanating from the objects;analyzing the plurality of images to identify at least onesound-emanating object in the environment of the user; retrieving from adatabase information about the at least one sound-emanating object;based on the retrieved information, causing selective conditioning of atleast one audio signal acquired by the wearable microphone from a regionassociated with the at least one sound-emanating object; causingtransmission of the at least one conditioned audio signal to a hearinginterface device configured to provide sounds to an ear of the user.

In an embodiment, a software product may be stored on a non-transitorycomputer readable medium and may comprise computer implementableinstructions for a method for identifying sound-emanating objects. Themethod may comprise accessing a database of reference visual signaturesand reference voice signatures corresponding to a plurality of objects;receiving a plurality of images captured by a wearable camera, whereinat least one of the plurality of images depicts at least onesound-emanating object in an environment of a user; analyzing thereceived at least one of the plurality of images to determine one ormore visual characteristics associated with the at least onesound-emanating object; identifying, based on review of the database inview of the one or more visual characteristics, the at least onesound-emanating object and determine a degree of certainty ofidentification; receiving audio signals acquired by a wearablemicrophone, wherein the audio signals are representative of one or moresounds emanating from the at least one sound-emanating object; analyzingthe received audio signals to determine voiceprint of the at least onesound-emanating object; when the degree of certainty of identificationfalls below a predetermined level, further identifying the at least onesound-emanating object based on the determined voiceprint; andinitiating at least one action based on an identity of the at least onesound-emanating object.

In an embodiment, a hearing interface device is disclosed. The hearinginterface device may comprise a receiver configured to receive at leastone audio signal, wherein the at least one audio signal was acquired bya wearable microphone and was selectively conditioned by at least oneprocessor configured to receive a plurality of images captured by awearable camera, identify at least one sound-emanating object in theplurality of images, and cause the conditioning based on retrievedinformation about the at least one sound-emanating object. The hearingaid device may further comprise an electroacoustic transducer configuredto provide sounds from the at least one audio signal to an ear of theuser.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processor and perform any of the methodsdescribed herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1A is a schematic illustration of an example of a user wearing awearable apparatus according to a disclosed embodiment.

FIG. 1B is a schematic illustration of an example of the user wearing awearable apparatus according to a disclosed embodiment.

FIG. 1C is a schematic illustration of an example of the user wearing awearable apparatus according to a disclosed embodiment.

FIG. 1D is a schematic illustration of an example of the user wearing awearable apparatus according to a disclosed embodiment.

FIG. 2 is a schematic illustration of an example system consistent withthe disclosed embodiments.

FIG. 3A is a schematic illustration of an example of the wearableapparatus shown in FIG. 1A.

FIG. 3B is an exploded view of the example of the wearable apparatusshown in FIG. 3A.

FIG. 4A-4K are schematic illustrations of an example of the wearableapparatus shown in FIG. 1B from various viewpoints.

FIG. 5A is a block diagram illustrating an example of the components ofa wearable apparatus according to a first embodiment.

FIG. 5B is a block diagram illustrating an example of the components ofa wearable apparatus according to a second embodiment.

FIG. 5C is a block diagram illustrating an example of the components ofa wearable apparatus according to a third embodiment.

FIG. 6 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIG. 7 is a schematic illustration of an embodiment of a wearableapparatus including an orientable image capture unit.

FIG. 8 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 9 is a schematic illustration of a user wearing a wearableapparatus consistent with an embodiment of the present disclosure.

FIG. 10 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 11 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 12 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 13 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 14 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 15 is a schematic illustration of an embodiment of a wearableapparatus power unit including a power source.

FIG. 16 is a schematic illustration of an exemplary embodiment of awearable apparatus including protective circuitry.

FIG. 17A is a schematic illustration of an example of a user wearing anapparatus for a camera-based hearing aid device according to a disclosedembodiment.

FIG. 17B is a schematic illustration of an embodiment of an apparatussecurable to an article of clothing consistent with the presentdisclosure.

FIG. 18 is a schematic illustration showing an exemplary environment foruse of a camera-based hearing aid consistent with the presentdisclosure.

FIG. 19 is a flowchart showing an exemplary process for selectivelyamplifying sounds emanating from a detected look direction of a userconsistent with disclosed embodiments.

FIG. 20A is a schematic illustration showing an exemplary environmentfor use of a hearing aid with voice and/or image recognition consistentwith the present disclosure.

FIG. 20B illustrates an exemplary embodiment of an apparatus comprisingfacial and voice recognition components consistent with the presentdisclosure.

FIG. 21 is a flowchart showing an exemplary process for selectivelyamplifying audio signals associated with a voice of a recognizedindividual consistent with disclosed embodiments.

FIG. 22 is a flowchart showing an exemplary process for selectivelytransmitting audio signals associated with a voice of a recognized userconsistent with disclosed embodiments.

FIG. 23A is a schematic illustration showing an exemplary individualthat may be identified in the environment of a user consistent with thepresent disclosure.

FIG. 23B is a schematic illustration showing an exemplary individualthat may be identified in the environment of a user consistent with thepresent disclosure.

FIG. 23C illustrates an exemplary lip-tracking system consistent withthe disclosed embodiments.

FIG. 24 is a schematic illustration showing an exemplary environment foruse of a lip-tracking hearing aid consistent with the presentdisclosure.

FIG. 25 is a flowchart showing an exemplary process for selectivelyamplifying audio signals based on tracked lip movements consistent withdisclosed embodiments.

FIG. 26 is a schematic illustration of an exemplary hearing aid systemconsistent with the present disclosure.

FIG. 27 is a schematic illustration of an exemplary image captured by animaging capture device consistent with the present disclosure.

FIG. 28 is a flowchart of an exemplary process for selectivelyamplifying an audio signal.

FIG. 29 is a schematic illustration of an exemplary hearing aid systemconsistent with the present disclosure.

FIGS. 30A and 30B are schematic illustrations of exemplary imagescaptured by an imaging capture device consistent with the presentdisclosure.

FIG. 31A is a flowchart of an exemplary process for selectivelyamplifying audio signals.

FIG. 31B is a flowchart of an exemplary process for selectivelyamplifying audio signals.

FIG. 31C is a flowchart of an exemplary process for selectivelyamplifying audio signals.

FIG. 32 is a schematic illustration of an example system including awearable apparatus according to a disclosed embodiment.

FIG. 33 is an example illustration of a user with a wearable devicecommunicating with other people according to a disclosed embodiment.

FIGS. 34A and 34B are example flowcharts describing a process ofisolating one or more voices of different speakers from an audio signalaccording to disclosed embodiments.

FIG. 35A is an example flowchart describing a process of separating avoice of a speaker from an audio signal according to disclosedembodiments.

FIG. 35B is an example flowchart describing a process of transmitting toa hearing device a conditioned audio signal according to disclosedembodiments.

FIG. 36A is an example flowchart describing a process of separating avoice of a speaker from an audio signal according to disclosedembodiments.

FIG. 36B is a block diagram of modules of a wearable apparatusconsistent with the disclosed embodiments.

FIGS. 37A-37C are example flowcharts describing a process oftransmitting to a hearing device a conditioned audio signal according todisclosed embodiments.

FIG. 38A is a block diagram illustrating a hearing aid system accordingto an example embodiment.

FIG. 38B is a schematic illustration showing an exemplary environmentfor use of a hearing aid with instruction deduction consistent with thepresent disclosure.

FIG. 38C is a schematic illustration of an exemplary hearing aid systemconsistent with the present disclosure.

FIGS. 39A and 39B are flowcharts illustrating processes for deducinginstructions for a hearing aid system according to a first embodiment.

FIGS. 40A and 40B are flowcharts illustrating processes for deducinginstructions for a hearing aid system according to a second embodiment.

FIG. 41A is a block chart illustrating an exemplary embodiment of amemory device containing software modules consistent with the presentdisclosure.

FIG. 41B is a schematic illustration showing an exemplary environment ofuser of a hearing aid system that selectively conditions audio signalsconsistent with the present disclosure.

FIGS. 42A-42F are schematic illustrations of audio signals acquired andprocessed by the hearing aid system illustrated in FIG. 41B consistentwith the present disclosure.

FIG. 43A is a flowchart showing an exemplary process for selectivelyconditioning audio signals associated with a recognized object,consistent with disclosed embodiments.

FIG. 43B is a flowchart showing another exemplary process forselectively conditioning audio signals associated with a recognizedobject, consistent with disclosed embodiments.

FIG. 44A is a schematic illustration showing an exemplary environment ofa user that includes sound-emanating objects responsible for backgroundnoises consistent with the present disclosure.

FIG. 44B is a schematic illustration of the audio signals acquired by awearable microphone in the scenario illustrated in FIG. 44A, consistentwith the present disclosure.

FIG. 44C is a schematic illustration of the conditioned audio signalstransmitted to a hearing interface device in the scenario illustrated inFIG. 44A, consistent with the present disclosure.

FIG. 45 is a block diagram illustrating an example of the components ofa hearing interface device consistent with the present disclosure.

FIG. 46A is a flowchart showing an exemplary process for selectivemodification of background noises based on determined importance levels,consistent with disclosed embodiments.

FIG. 46B is a flowchart showing an exemplary process for selectivemodification of background noises, consistent with disclosedembodiments.

FIG. 47A is a block diagram illustrating an exemplary hearing aid systemconsistent with the present disclosure.

FIG. 47B is a schematic illustration showing an exemplary environmentfor using voice and visual signatures to identify objects consistentwith the present disclosure.

FIG. 48 is an illustration showing an exemplary device displaying thename of a sound emanating object with the present disclosure.

FIG. 49 is a flowchart showing an exemplary process for using voice andvisual signatures to identify objects consistent with disclosedembodiments.

FIG. 50A is a schematic illustration showing examples of sound emittingobjects that may be identified in the environment of a user consistentwith the present disclosure.

FIG. 50B is an illustration of an example database storing informationassociated with sound emanating objects consistent with the presentdisclosure.

FIGS. 51A and 51B are schematic illustrations showing exampleenvironments for selectively conditioning audio signals consistent withthe present disclosure.

FIG. 52 is a flowchart showing an exemplary process for modifying soundsemanating from objects in an environment of a user consistent with thedisclosed embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

FIG. 1A illustrates a user 100 wearing an apparatus 110 that isphysically connected (or integral) to glasses 130, consistent with thedisclosed embodiments. Glasses 130 may be prescription glasses,magnifying glasses, non-prescription glasses, safety glasses,sunglasses, etc. Additionally, in some embodiments, glasses 130 mayinclude parts of a frame and earpieces, nosepieces, etc., and one or nolenses. Thus, in some embodiments, glasses 130 may function primarily tosupport apparatus 110, and/or an augmented reality display device orother optical display device. In some embodiments, apparatus 110 mayinclude an image sensor (not shown in FIG. 1A) for capturing real-timeimage data of the field-of-view of user 100. The term “image data”includes any form of data retrieved from optical signals in thenear-infrared, infrared, visible, and ultraviolet spectrums. The imagedata may include video clips and/or photographs.

In some embodiments, apparatus 110 may communicate wirelessly or via awire with a computing device 120. In some embodiments, computing device120 may include, for example, a smartphone, or a tablet, or a dedicatedprocessing unit, which may be portable (e.g., can be carried in a pocketof user 100). Although shown in FIG. 1A as an external device, in someembodiments, computing device 120 may be provided as part of wearableapparatus 110 or glasses 130, whether integral thereto or mountedthereon. In some embodiments, computing device 120 may be included in anaugmented reality display device or optical head mounted displayprovided integrally or mounted to glasses 130. In other embodiments,computing device 120 may be provided as part of another wearable orportable apparatus of user 100 including a wrist-strap, amultifunctional watch, a button, a clip-on, etc. And in otherembodiments, computing device 120 may be provided as part of anothersystem, such as an on-board automobile computing or navigation system. Aperson skilled in the art can appreciate that different types ofcomputing devices and arrangements of devices may implement thefunctionality of the disclosed embodiments. Accordingly, in otherimplementations, computing device 120 may include a Personal Computer(PC), laptop, an Internet server, etc.

FIG. 1B illustrates user 100 wearing apparatus 110 that is physicallyconnected to a necklace 140, consistent with a disclosed embodiment.Such a configuration of apparatus 110 may be suitable for users that donot wear glasses some or all of the time. In this embodiment, user 100can easily wear apparatus 110, and take it off.

FIG. 1C illustrates user 100 wearing apparatus 110 that is physicallyconnected to a belt 150, consistent with a disclosed embodiment. Such aconfiguration of apparatus 110 may be designed as a belt buckle.Alternatively, apparatus 110 may include a clip for attaching to variousclothing articles, such as belt 150, or a vest, a pocket, a collar, acap or hat or other portion of a clothing article.

FIG. 1D illustrates user 100 wearing apparatus 110 that is physicallyconnected to a wrist strap 160, consistent with a disclosed embodiment.Although the aiming direction of apparatus 110, according to thisembodiment, may not match the field-of-view of user 100, apparatus 110may include the ability to identify a hand-related trigger based on thetracked eye movement of a user 100 indicating that user 100 is lookingin the direction of the wrist strap 160. Wrist strap 160 may alsoinclude an accelerometer, a gyroscope, or other sensor for determiningmovement or orientation of a user's 100 hand for identifying ahand-related trigger.

FIG. 2 is a schematic illustration of an exemplary system 200 includinga wearable apparatus 110, worn by user 100, and an optional computingdevice 120 and/or a server 250 capable of communicating with apparatus110 via a network 240, consistent with disclosed embodiments. In someembodiments, apparatus 110 may capture and analyze image data, identifya hand-related trigger present in the image data, and perform an actionand/or provide feedback to a user 100, based at least in part on theidentification of the hand-related trigger. In some embodiments,optional computing device 120 and/or server 250 may provide additionalfunctionality to enhance interactions of user 100 with his or herenvironment, as described in greater detail below.

According to the disclosed embodiments, apparatus 110 may include animage sensor system 220 for capturing real-time image data of thefield-of-view of user 100. In some embodiments, apparatus 110 may alsoinclude a processing unit 210 for controlling and performing thedisclosed functionality of apparatus 110, such as to control the captureof image data, analyze the image data, and perform an action and/oroutput a feedback based on a hand-related trigger identified in theimage data. According to the disclosed embodiments, a hand-relatedtrigger may include a gesture performed by user 100 involving a portionof a hand of user 100. Further, consistent with some embodiments, ahand-related trigger may include a wrist-related trigger. Additionally,in some embodiments, apparatus 110 may include a feedback outputtingunit 230 for producing an output of information to user 100.

As discussed above, apparatus 110 may include an image sensor 220 forcapturing image data. The term “image sensor” refers to a device capableof detecting and converting optical signals in the near-infrared,infrared, visible, and ultraviolet spectrums into electrical signals.The electrical signals may be used to form an image or a video stream(i.e. image data) based on the detected signal. The term “image data”includes any form of data retrieved from optical signals in thenear-infrared, infrared, visible, and ultraviolet spectrums. Examples ofimage sensors may include semiconductor charge-coupled devices (CCD),active pixel sensors in complementary metal-oxide-semiconductor (CMOS),or N-type metal-oxide-semiconductor (NMOS, Live MOS). In some cases,image sensor 220 may be part of a camera included in apparatus 110.

Apparatus 110 may also include a processor 210 for controlling imagesensor 220 to capture image data and for analyzing the image dataaccording to the disclosed embodiments. As discussed in further detailbelow with respect to FIG. 5A, processor 210 may include a “processingdevice” for performing logic operations on one or more inputs of imagedata and other data according to stored or accessible softwareinstructions providing desired functionality. In some embodiments,processor 210 may also control feedback outputting unit 230 to providefeedback to user 100 including information based on the analyzed imagedata and the stored software instructions. As the term is used herein, a“processing device” may access memory where executable instructions arestored or, in some embodiments, a “processing device” itself may includeexecutable instructions (e.g., stored in memory included in theprocessing device).

In some embodiments, the information or feedback information provided touser 100 may include time information. The time information may includeany information related to a current time of day and, as describedfurther below, may be presented in any sensory perceptive manner. Insome embodiments, time information may include a current time of day ina preconfigured format (e.g., 2:30 pm or 14:30). Time information mayinclude the time in the user's current time zone (e.g., based on adetermined location of user 100), as well as an indication of the timezone and/or a time of day in another desired location. In someembodiments, time information may include a number of hours or minutesrelative to one or more predetermined times of day. For example, in someembodiments, time information may include an indication that three hoursand fifteen minutes remain until a particular hour (e.g., until 6:00pm), or some other predetermined time. Time information may also includea duration of time passed since the beginning of a particular activity,such as the start of a meeting or the start of a jog, or any otheractivity. In some embodiments, the activity may be determined based onanalyzed image data. In other embodiments, time information may alsoinclude additional information related to a current time and one or moreother routine, periodic, or scheduled events. For example, timeinformation may include an indication of the number of minutes remaininguntil the next scheduled event, as may be determined from a calendarfunction or other information retrieved from computing device 120 orserver 250, as discussed in further detail below.

Feedback outputting unit 230 may include one or more feedback systemsfor providing the output of information to user 100. In the disclosedembodiments, the audible or visual feedback may be provided via any typeof connected audible or visual system or both. Feedback of informationaccording to the disclosed embodiments may include audible feedback touser 100 (e.g., using a Bluetooth™ or other wired or wirelesslyconnected speaker, or a bone conduction headphone). Feedback outputtingunit 230 of some embodiments may additionally or alternatively produce avisible output of information to user 100, for example, as part of anaugmented reality display projected onto a lens of glasses 130 orprovided via a separate heads up display in communication with apparatus110, such as a display 260 provided as part of computing device 120,which may include an onboard automobile heads up display, an augmentedreality device, a virtual reality device, a smartphone, PC, table, etc.

The term “computing device” refers to a device including a processingunit and having computing capabilities. Some examples of computingdevice 120 include a PC, laptop, tablet, or other computing systems suchas an on-board computing system of an automobile, for example, eachconfigured to communicate directly with apparatus 110 or server 250 overnetwork 240. Another example of computing device 120 includes asmartphone having a display 260. In some embodiments, computing device120 may be a computing system configured particularly for apparatus 110,and may be provided integral to apparatus 110 or tethered thereto.Apparatus 110 can also connect to computing device 120 over network 240via any known wireless standard (e.g., Wi-Fi, Bluetooth®, etc.), as wellas near-filed capacitive coupling, and other short range wirelesstechniques, or via a wired connection. In an embodiment in whichcomputing device 120 is a smartphone, computing device 120 may have adedicated application installed therein. For example, user 100 may viewon display 260 data (e.g., images, video clips, extracted information,feedback information, etc.) that originate from or are triggered byapparatus 110. In addition, user 100 may select part of the data forstorage in server 250.

Network 240 may be a shared, public, or private network, may encompass awide area or local area, and may be implemented through any suitablecombination of wired and/or wireless communication networks. Network 240may further comprise an intranet or the Internet. In some embodiments,network 240 may include short range or near-field wireless communicationsystems for enabling communication between apparatus 110 and computingdevice 120 provided in close proximity to each other, such as on or neara user's person, for example. Apparatus 110 may establish a connectionto network 240 autonomously, for example, using a wireless module (e.g.,Wi-Fi, cellular). In some embodiments, apparatus 110 may use thewireless module when being connected to an external power source, toprolong battery life. Further, communication between apparatus 110 andserver 250 may be accomplished through any suitable communicationchannels, such as, for example, a telephone network, an extranet, anintranet, the Internet, satellite communications, off-linecommunications, wireless communications, transponder communications, alocal area network (LAN), a wide area network (WAN), and a virtualprivate network (VPN).

As shown in FIG. 2, apparatus 110 may transfer or receive data to/fromserver 250 via network 240. In the disclosed embodiments, the data beingreceived from server 250 and/or computing device 120 may includenumerous different types of information based on the analyzed imagedata, including information related to a commercial product, or aperson's identity, an identified landmark, and any other informationcapable of being stored in or accessed by server 250. In someembodiments, data may be received and transferred via computing device120. Server 250 and/or computing device 120 may retrieve informationfrom different data sources (e.g., a user specific database or a user'ssocial network account or other account, the Internet, and other managedor accessible databases) and provide information to apparatus 110related to the analyzed image data and a recognized trigger according tothe disclosed embodiments. In some embodiments, calendar-relatedinformation retrieved from the different data sources may be analyzed toprovide certain time information or a time-based context for providingcertain information based on the analyzed image data.

An example of wearable apparatus 110 incorporated with glasses 130according to some embodiments (as discussed in connection with FIG. 1A)is shown in greater detail in FIG. 3A. In some embodiments, apparatus110 may be associated with a structure (not shown in FIG. 3A) thatenables easy detaching and reattaching of apparatus 110 to glasses 130.In some embodiments, when apparatus 110 attaches to glasses 130, imagesensor 220 acquires a set aiming direction without the need fordirectional calibration. The set aiming direction of image sensor 220may substantially coincide with the field-of-view of user 100. Forexample, a camera associated with image sensor 220 may be installedwithin apparatus 110 in a predetermined angle in a position facingslightly downwards (e.g., 5-15 degrees from the horizon). Accordingly,the set aiming direction of image sensor 220 may substantially match thefield-of-view of user 100.

FIG. 3B is an exploded view of the components of the embodimentdiscussed regarding FIG. 3A. Attaching apparatus 110 to glasses 130 maytake place in the following way. Initially, a support 310 may be mountedon glasses 130 using a screw 320, in the side of support 310. Then,apparatus 110 may be clipped on support 310 such that it is aligned withthe field-of-view of user 100. The term “support” includes any device orstructure that enables detaching and reattaching of a device including acamera to a pair of glasses or to another object (e.g., a helmet).Support 310 may be made from plastic (e.g., polycarbonate), metal (e.g.,aluminum), or a combination of plastic and metal (e.g., carbon fibergraphite). Support 310 may be mounted on any kind of glasses (e.g.,eyeglasses, sunglasses, 3D glasses, safety glasses, etc.) using screws,bolts, snaps, or any fastening means used in the art.

In some embodiments, support 310 may include a quick release mechanismfor disengaging and reengaging apparatus 110. For example, support 310and apparatus 110 may include magnetic elements. As an alternativeexample, support 310 may include a male latch member and apparatus 110may include a female receptacle. In other embodiments, support 310 canbe an integral part of a pair of glasses, or sold separately andinstalled by an optometrist. For example, support 310 may be configuredfor mounting on the arms of glasses 130 near the frame front, but beforethe hinge. Alternatively, support 310 may be configured for mounting onthe bridge of glasses 130.

In some embodiments, apparatus 110 may be provided as part of a glassesframe 130, with or without lenses. Additionally, in some embodiments,apparatus 110 may be configured to provide an augmented reality displayprojected onto a lens of glasses 130 (if provided), or alternatively,may include a display for projecting time information, for example,according to the disclosed embodiments. Apparatus 110 may include theadditional display or alternatively, may be in communication with aseparately provided display system that may or may not be attached toglasses 130.

In some embodiments, apparatus 110 may be implemented in a form otherthan wearable glasses, as described above with respect to FIGS. 1B-1D,for example. FIG. 4A is a schematic illustration of an example of anadditional embodiment of apparatus 110 from a front viewpoint ofapparatus 110. Apparatus 110 includes an image sensor 220, a clip (notshown), a function button (not shown) and a hanging ring 410 forattaching apparatus 110 to, for example, necklace 140, as shown in FIG.1B. When apparatus 110 hangs on necklace 140, the aiming direction ofimage sensor 220 may not fully coincide with the field-of-view of user100, but the aiming direction would still correlate with thefield-of-view of user 100.

FIG. 4B is a schematic illustration of the example of a secondembodiment of apparatus 110, from a side orientation of apparatus 110.In addition to hanging ring 410, as shown in FIG. 4B, apparatus 110 mayfurther include a clip 420. User 100 can use clip 420 to attachapparatus 110 to a shirt or belt 150, as illustrated in FIG. 1C. Clip420 may provide an easy mechanism for disengaging and re-engagingapparatus 110 from different articles of clothing. In other embodiments,apparatus 110 may include a female receptacle for connecting with a malelatch of a car mount or universal stand.

In some embodiments, apparatus 110 includes a function button 430 forenabling user 100 to provide input to apparatus 110. Function button 430may accept different types of tactile input (e.g., a tap, a click, adouble-click, a long press, a right-to-left slide, a left-to-rightslide). In some embodiments, each type of input may be associated with adifferent action. For example, a tap may be associated with the functionof taking a picture, while a right-to-left slide may be associated withthe function of recording a video.

Apparatus 110 may be attached to an article of clothing (e.g., a shirt,a belt, pants, etc.), of user 100 at an edge of the clothing using aclip 431 as shown in FIG. 4C. For example, the body of apparatus 100 mayreside adjacent to the inside surface of the clothing with clip 431engaging with the outside surface of the clothing. In such anembodiment, as shown in FIG. 4C, the image sensor 220 (e.g., a camerafor visible light) may be protruding beyond the edge of the clothing.Alternatively, clip 431 may be engaging with the inside surface of theclothing with the body of apparatus 110 being adjacent to the outside ofthe clothing. In various embodiments, the clothing may be positionedbetween clip 431 and the body of apparatus 110.

An example embodiment of apparatus 110 is shown in FIG. 4D. Apparatus110 includes clip 431 which may include points (e.g., 432A and 432B) inclose proximity to a front surface 434 of a body 435 of apparatus 110.In an example embodiment, the distance between points 432A, 432B andfront surface 434 may be less than a typical thickness of a fabric ofthe clothing of user 100. For example, the distance between points 432A,432B and surface 434 may be less than a thickness of a tee-shirt, e.g.,less than a millimeter, less than 2 millimeters, less than 3millimeters, etc., or, in some cases, points 432A, 432B of clip 431 maytouch surface 434. In various embodiments, clip 431 may include a point433 that does not touch surface 434, allowing the clothing to beinserted between clip 431 and surface 434.

FIG. 4D shows schematically different views of apparatus 110 defined asa front view (F-view), a rearview (R-view), a top view (T-view), a sideview (S-view) and a bottom view (B-view). These views will be referredto when describing apparatus 110 in subsequent figures. FIG. 4D shows anexample embodiment where clip 431 is positioned at the same side ofapparatus 110 as sensor 220 (e.g., the front side of apparatus 110).Alternatively, clip 431 may be positioned at an opposite side ofapparatus 110 as sensor 220 (e.g., the rear side of apparatus 110). Invarious embodiments, apparatus 110 may include function button 430, asshown in FIG. 4D.

Various views of apparatus 110 are illustrated in FIGS. 4E through 4K.For example, FIG. 4E shows a view of apparatus 110 with an electricalconnection 441. Electrical connection 441 may be, for example, a USBport, that may be used to transfer data to/from apparatus 110 andprovide electrical power to apparatus 110. In an example embodiment,connection 441 may be used to charge a battery 442 schematically shownin FIG. 4E. FIG. 4F shows F-view of apparatus 110, including sensor 220and one or more microphones 443. In some embodiments, apparatus 110 mayinclude several microphones 443 facing outwards, wherein microphones 443are configured to obtain environmental sounds and sounds of variousspeakers communicating with user 100. FIG. 4G shows R-view of apparatus110. In some embodiments, microphone 444 may be positioned at the rearside of apparatus 110, as shown in FIG. 4G. Microphone 444 may be usedto detect an audio signal from user 100. It should be noted, thatapparatus 110 may have microphones placed at any side (e.g., a frontside, a rear side, a left side, a right side, a top side, or a bottomside) of apparatus 110. In various embodiments, some microphones may beat a first side (e.g., microphones 443 may be at the front of apparatus110) and other microphones may be at a second side (e.g., microphone 444may be at the back side of apparatus 110).

FIGS. 4H and 4I show different sides of apparatus 110 (i.e., S-view ofapparatus 110) consisted with disclosed embodiments. For example, FIG.4H shows the location of sensor 220 and an example shape of clip 431.FIG. 4J shows T-view of apparatus 110, including function button 430,and FIG. 4K shows B-view of apparatus 110 with electrical connection441.

The example embodiments discussed above with respect to FIGS. 3A, 3B,4A, and 4B are not limiting. In some embodiments, apparatus 110 may beimplemented in any suitable configuration for performing the disclosedmethods. For example, referring back to FIG. 2, the disclosedembodiments may implement an apparatus 110 according to anyconfiguration including an image sensor 220 and a processor unit 210 toperform image analysis and for communicating with a feedback unit 230.

FIG. 5A is a block diagram illustrating the components of apparatus 110according to an example embodiment. As shown in FIG. 5A, and assimilarly discussed above, apparatus 110 includes an image sensor 220, amemory 550, a processor 210, a feedback outputting unit 230, a wirelesstransceiver 530, and a mobile power source 520. In other embodiments,apparatus 110 may also include buttons, other sensors such as amicrophone, and inertial measurements devices such as accelerometers,gyroscopes, magnetometers, temperature sensors, color sensors, lightsensors, etc. Apparatus 110 may further include a data port 570 and apower connection 510 with suitable interfaces for connecting with anexternal power source or an external device (not shown).

Processor 210, depicted in FIG. 5A, may include any suitable processingdevice. The term “processing device” includes any physical device havingan electric circuit that performs a logic operation on input or inputs.For example, processing device may include one or more integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field-programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. The instructions executed by the processing device may, forexample, be pre-loaded into a memory integrated with or embedded intothe processing device or may be stored in a separate memory (e.g.,memory 550). Memory 550 may comprise a Random Access Memory (RAM), aRead-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium,a flash memory, other permanent, fixed, or volatile memory, or any othermechanism capable of storing instructions.

Although, in the embodiment illustrated in FIG. 5A, apparatus 110includes one processing device (e.g., processor 210), apparatus 110 mayinclude more than one processing device. Each processing device may havea similar construction, or the processing devices may be of differingconstructions that are electrically connected or disconnected from eachother. For example, the processing devices may be separate circuits orintegrated in a single circuit. When more than one processing device isused, the processing devices may be configured to operate independentlyor collaboratively. The processing devices may be coupled electrically,magnetically, optically, acoustically, mechanically or by other meansthat permit them to interact.

In some embodiments, processor 210 may process a plurality of imagescaptured from the environment of user 100 to determine differentparameters related to capturing subsequent images. For example,processor 210 can determine, based on information derived from capturedimage data, a value for at least one of the following: an imageresolution, a compression ratio, a cropping parameter, frame rate, afocus point, an exposure time, an aperture size, and a lightsensitivity. The determined value may be used in capturing at least onesubsequent image. Additionally, processor 210 can detect imagesincluding at least one hand-related trigger in the environment of theuser and perform an action and/or provide an output of information to auser via feedback outputting unit 230.

In another embodiment, processor 210 can change the aiming direction ofimage sensor 220. For example, when apparatus 110 is attached with clip420, the aiming direction of image sensor 220 may not coincide with thefield-of-view of user 100. Processor 210 may recognize certainsituations from the analyzed image data and adjust the aiming directionof image sensor 220 to capture relevant image data. For example, in oneembodiment, processor 210 may detect an interaction with anotherindividual and sense that the individual is not fully in view, becauseimage sensor 220 is tilted down. Responsive thereto, processor 210 mayadjust the aiming direction of image sensor 220 to capture image data ofthe individual. Other scenarios are also contemplated where processor210 may recognize the need to adjust an aiming direction of image sensor220.

In some embodiments, processor 210 may communicate data tofeedback-outputting unit 230, which may include any device configured toprovide information to a user 100. Feedback outputting unit 230 may beprovided as part of apparatus 110 (as shown) or may be provided externalto apparatus 110 and communicatively coupled thereto.Feedback-outputting unit 230 may be configured to output visual ornonvisual feedback based on signals received from processor 210, such aswhen processor 210 recognizes a hand-related trigger in the analyzedimage data.

The term “feedback” refers to any output or information provided inresponse to processing at least one image in an environment. In someembodiments, as similarly described above, feedback may include anaudible or visible indication of time information, detected text ornumerals, the value of currency, a branded product, a person's identity,the identity of a landmark or other environmental situation or conditionincluding the street names at an intersection or the color of a trafficlight, etc., as well as other information associated with each of these.For example, in some embodiments, feedback may include additionalinformation regarding the amount of currency still needed to complete atransaction, information regarding the identified person, historicalinformation or times and prices of admission etc. of a detected landmarketc. In some embodiments, feedback may include an audible tone, atactile response, and/or information previously recorded by user 100.Feedback-outputting unit 230 may comprise appropriate components foroutputting acoustical and tactile feedback. For example,feedback-outputting unit 230 may comprise audio headphones, a hearingaid type device, a speaker, a bone conduction headphone, interfaces thatprovide tactile cues, vibrotactile stimulators, etc. In someembodiments, processor 210 may communicate signals with an externalfeedback outputting unit 230 via a wireless transceiver 530, a wiredconnection, or some other communication interface. In some embodiments,feedback outputting unit 230 may also include any suitable displaydevice for visually displaying information to user 100.

As shown in FIG. 5A, apparatus 110 includes memory 550. Memory 550 mayinclude one or more sets of instructions accessible to processor 210 toperform the disclosed methods, including instructions for recognizing ahand-related trigger in the image data. In some embodiments memory 550may store image data (e.g., images, videos) captured from theenvironment of user 100. In addition, memory 550 may store informationspecific to user 100, such as image representations of knownindividuals, favorite products, personal items, and calendar orappointment information, etc. In some embodiments, processor 210 maydetermine, for example, which type of image data to store based onavailable storage space in memory 550. In another embodiment, processor210 may extract information from the image data stored in memory 550.

As further shown in FIG. 5A, apparatus 110 includes mobile power source520. The term “mobile power source” includes any device capable ofproviding electrical power, which can be easily carried by hand (e.g.,mobile power source 520 may weigh less than a pound). The mobility ofthe power source enables user 100 to use apparatus 110 in a variety ofsituations. In some embodiments, mobile power source 520 may include oneor more batteries (e.g., nickel-cadmium batteries, nickel-metal hydridebatteries, and lithium-ion batteries) or any other type of electricalpower supply. In other embodiments, mobile power source 520 may berechargeable and contained within a casing that holds apparatus 110. Inyet other embodiments, mobile power source 520 may include one or moreenergy harvesting devices for converting ambient energy into electricalenergy (e.g., portable solar power units, human vibration units, etc.).

Mobile power source 520 may power one or more wireless transceivers(e.g., wireless transceiver 530 in FIG. 5A). The term “wirelesstransceiver” refers to any device configured to exchange transmissionsover an air interface by use of radio frequency, infrared frequency,magnetic field, or electric field. Wireless transceiver 530 may use anyknown standard to transmit and/or receive data (e.g., Wi-Fi, Bluetooth®,Bluetooth Smart, 802.15.4, or ZigBee). In some embodiments, wirelesstransceiver 530 may transmit data (e.g., raw image data, processed imagedata, extracted information) from apparatus 110 to computing device 120and/or server 250. Wireless transceiver 530 may also receive data fromcomputing device 120 and/or server 250. In other embodiments, wirelesstransceiver 530 may transmit data and instructions to an externalfeedback outputting unit 230.

FIG. 5B is a block diagram illustrating the components of apparatus 110according to another example embodiment. In some embodiments, apparatus110 includes a first image sensor 220 a, a second image sensor 220 b, amemory 550, a first processor 210 a, a second processor 210 b, afeedback outputting unit 230, a wireless transceiver 530, a mobile powersource 520, and a power connector 510. In the arrangement shown in FIG.5B, each of the image sensors may provide images in a different imageresolution, or face a different direction. Alternatively, each imagesensor may be associated with a different camera (e.g., a wide anglecamera, a narrow angle camera, an IR camera, etc.). In some embodiments,apparatus 110 can select which image sensor to use based on variousfactors. For example, processor 210 a may determine, based on availablestorage space in memory 550, to capture subsequent images in a certainresolution.

Apparatus 110 may operate in a first processing-mode and in a secondprocessing-mode, such that the first processing-mode may consume lesspower than the second processing-mode. For example, in the firstprocessing-mode, apparatus 110 may capture images and process thecaptured images to make real-time decisions based on an identifyinghand-related trigger, for example. In the second processing-mode,apparatus 110 may extract information from stored images in memory 550and delete images from memory 550. In some embodiments, mobile powersource 520 may provide more than fifteen hours of processing in thefirst processing-mode and about three hours of processing in the secondprocessing-mode. Accordingly, different processing-modes may allowmobile power source 520 to produce sufficient power for poweringapparatus 110 for various time periods (e.g., more than two hours, morethan four hours, more than ten hours, etc.).

In some embodiments, apparatus 110 may use first processor 210 a in thefirst processing-mode when powered by mobile power source 520, andsecond processor 210 b in the second processing-mode when powered byexternal power source 580 that is connectable via power connector 510.In other embodiments, apparatus 110 may determine, based on predefinedconditions, which processors or which processing modes to use. Apparatus110 may operate in the second processing-mode even when apparatus 110 isnot powered by external power source 580. For example, apparatus 110 maydetermine that it should operate in the second processing-mode whenapparatus 110 is not powered by external power source 580, if theavailable storage space in memory 550 for storing new image data islower than a predefined threshold.

Although one wireless transceiver is depicted in FIG. 5B, apparatus 110may include more than one wireless transceiver (e.g., two wirelesstransceivers). In an arrangement with more than one wirelesstransceiver, each of the wireless transceivers may use a differentstandard to transmit and/or receive data. In some embodiments, a firstwireless transceiver may communicate with server 250 or computing device120 using a cellular standard (e.g., LTE or GSM), and a second wirelesstransceiver may communicate with server 250 or computing device 120using a short-range standard (e.g., Wi-Fi or Bluetooth®). In someembodiments, apparatus 110 may use the first wireless transceiver whenthe wearable apparatus is powered by a mobile power source included inthe wearable apparatus, and use the second wireless transceiver when thewearable apparatus is powered by an external power source.

FIG. 5C is a block diagram illustrating the components of apparatus 110according to another example embodiment including computing device 120.In this embodiment, apparatus 110 includes an image sensor 220, a memory550 a, a first processor 210, a feedback-outputting unit 230, a wirelesstransceiver 530 a, a mobile power source 520, and a power connector 510.As further shown in FIG. 5C, computing device 120 includes a processor540, a feedback-outputting unit 545, a memory 550 b, a wirelesstransceiver 530 b, and a display 260. One example of computing device120 is a smartphone or tablet having a dedicated application installedtherein. In other embodiments, computing device 120 may include anyconfiguration such as an on-board automobile computing system, a PC, alaptop, and any other system consistent with the disclosed embodiments.In this example, user 100 may view feedback output in response toidentification of a hand-related trigger on display 260. Additionally,user 100 may view other data (e.g., images, video clips, objectinformation, schedule information, extracted information, etc.) ondisplay 260. In addition, user 100 may communicate with server 250 viacomputing device 120.

In some embodiments, processor 210 and processor 540 are configured toextract information from captured image data. The term “extractinginformation” includes any process by which information associated withobjects, individuals, locations, events, etc., is identified in thecaptured image data by any means known to those of ordinary skill in theart. In some embodiments, apparatus 110 may use the extractedinformation to send feedback or other real-time indications to feedbackoutputting unit 230 or to computing device 120. In some embodiments,processor 210 may identify in the image data the individual standing infront of user 100, and send computing device 120 the name of theindividual and the last time user 100 met the individual. In anotherembodiment, processor 210 may identify in the image data, one or morevisible triggers, including a hand-related trigger, and determinewhether the trigger is associated with a person other than the user ofthe wearable apparatus to selectively determine whether to perform anaction associated with the trigger. One such action may be to provide afeedback to user 100 via feedback-outputting unit 230 provided as partof (or in communication with) apparatus 110 or via a feedback unit 545provided as part of computing device 120. For example,feedback-outputting unit 545 may be in communication with display 260 tocause the display 260 to visibly output information. In someembodiments, processor 210 may identify in the image data a hand-relatedtrigger and send computing device 120 an indication of the trigger.Processor 540 may then process the received trigger information andprovide an output via feedback outputting unit 545 or display 260 basedon the hand-related trigger. In other embodiments, processor 540 maydetermine a hand-related trigger and provide suitable feedback similarto the above, based on image data received from apparatus 110. In someembodiments, processor 540 may provide instructions or otherinformation, such as environmental information to apparatus 110 based onan identified hand-related trigger.

In some embodiments, processor 210 may identify other environmentalinformation in the analyzed images, such as an individual standing infront user 100, and send computing device 120 information related to theanalyzed information such as the name of the individual and the lasttime user 100 met the individual. In a different embodiment, processor540 may extract statistical information from captured image data andforward the statistical information to server 250. For example, certaininformation regarding the types of items a user purchases, or thefrequency a user patronizes a particular merchant, etc. may bedetermined by processor 540. Based on this information, server 250 maysend computing device 120 coupons and discounts associated with theuser's preferences.

When apparatus 110 is connected or wirelessly connected to computingdevice 120, apparatus 110 may transmit at least part of the image datastored in memory 550 a for storage in memory 550 b. In some embodiments,after computing device 120 confirms that transferring the part of imagedata was successful, processor 540 may delete the part of the imagedata. The term “delete” means that the image is marked as ‘deleted’ andother image data may be stored instead of it, but does not necessarilymean that the image data was physically removed from the memory.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the disclosed embodiments. Not all components are essential for theoperation of apparatus 110. Any component may be located in anyappropriate apparatus and the components may be rearranged into avariety of configurations while providing the functionality of thedisclosed embodiments. For example, in some embodiments, apparatus 110may include a camera, a processor, and a wireless transceiver forsending data to another device. Therefore, the foregoing configurationsare examples and, regardless of the configurations discussed above,apparatus 110 can capture, store, and/or process images.

Further, the foregoing and following description refers to storingand/or processing images or image data. In the embodiments disclosedherein, the stored and/or processed images or image data may comprise arepresentation of one or more images captured by image sensor 220. Asthe term is used herein, a “representation” of an image (or image data)may include an entire image or a portion of an image. A representationof an image (or image data) may have the same resolution or a lowerresolution as the image (or image data), and/or a representation of animage (or image data) may be altered in some respect (e.g., becompressed, have a lower resolution, have one or more colors that arealtered, etc.).

For example, apparatus 110 may capture an image and store arepresentation of the image that is compressed as a .JPG file. Asanother example, apparatus 110 may capture an image in color, but storea black-and-white representation of the color image. As yet anotherexample, apparatus 110 may capture an image and store a differentrepresentation of the image (e.g., a portion of the image) For example,apparatus 110 may store a portion of an image that includes a face of aperson who appears in the image, but that does not substantially includethe environment surrounding the person. Similarly, apparatus 110 may,for example, store a portion of an image that includes a product thatappears in the image, but does not substantially include the environmentsurrounding the product. As yet another example, apparatus 110 may storea representation of an image at a reduced resolution (i.e., at aresolution that is of a lower value than that of the captured image).Storing representations of images may allow apparatus 110 to savestorage space in memory 550. Furthermore, processing representations ofimages may allow apparatus 110 to improve processing efficiency and/orhelp to preserve battery life.

In addition to the above, in some embodiments, any one of apparatus 110or computing device 120, via processor 210 or 540, may further processthe captured image data to provide additional functionality to recognizeobjects and/or gestures and/or other information in the captured imagedata. In some embodiments, actions may be taken based on the identifiedobjects, gestures, or other information. In some embodiments, processor210 or 540 may identify in the image data, one or more visible triggers,including a hand-related trigger, and determine whether the trigger isassociated with a person other than the user to determine whether toperform an action associated with the trigger.

Some embodiments of the present disclosure may include an apparatussecurable to an article of clothing of a user. Such an apparatus mayinclude two portions, connectable by a connector. A capturing unit maybe designed to be worn on the outside of a user's clothing, and mayinclude an image sensor for capturing images of a user's environment.The capturing unit may be connected to or connectable to a power unit,which may be configured to house a power source and a processing device.The capturing unit may be a small device including a camera or otherdevice for capturing images. The capturing unit may be designed to beinconspicuous and unobtrusive, and may be configured to communicate witha power unit concealed by a user's clothing. The power unit may includebulkier aspects of the system, such as transceiver antennas, at leastone battery, a processing device, etc. In some embodiments,communication between the capturing unit and the power unit may beprovided by a data cable included in the connector, while in otherembodiments, communication may be wirelessly achieved between thecapturing unit and the power unit. Some embodiments may permitalteration of the orientation of an image sensor of the capture unit,for example to better capture images of interest.

FIG. 6 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure. Included inmemory 550 are orientation identification module 601, orientationadjustment module 602, and motion tracking module 603. Modules 601, 602,603 may contain software instructions for execution by at least oneprocessing device, e.g., processor 210, included with a wearableapparatus. Orientation identification module 601, orientation adjustmentmodule 602, and motion tracking module 603 may cooperate to provideorientation adjustment for a capturing unit incorporated into wirelessapparatus 110.

FIG. 7 illustrates an exemplary capturing unit 710 including anorientation adjustment unit 705. Orientation adjustment unit 705 may beconfigured to permit the adjustment of image sensor 220. As illustratedin FIG. 7, orientation adjustment unit 705 may include an eye-ball typeadjustment mechanism. In alternative embodiments, orientation adjustmentunit 705 may include gimbals, adjustable stalks, pivotable mounts, andany other suitable unit for adjusting an orientation of image sensor220.

Image sensor 220 may be configured to be movable with the head of user100 in such a manner that an aiming direction of image sensor 220substantially coincides with a field of view of user 100. For example,as described above, a camera associated with image sensor 220 may beinstalled within capturing unit 710 at a predetermined angle in aposition facing slightly upwards or downwards, depending on an intendedlocation of capturing unit 710. Accordingly, the set aiming direction ofimage sensor 220 may match the field-of-view of user 100. In someembodiments, processor 210 may change the orientation of image sensor220 using image data provided from image sensor 220. For example,processor 210 may recognize that a user is reading a book and determinethat the aiming direction of image sensor 220 is offset from the text.That is, because the words in the beginning of each line of text are notfully in view, processor 210 may determine that image sensor 220 istilted in the wrong direction. Responsive thereto, processor 210 mayadjust the aiming direction of image sensor 220.

Orientation identification module 601 may be configured to identify anorientation of an image sensor 220 of capturing unit 710. An orientationof an image sensor 220 may be identified, for example, by analysis ofimages captured by image sensor 220 of capturing unit 710, by tilt orattitude sensing devices within capturing unit 710, and by measuring arelative direction of orientation adjustment unit 705 with respect tothe remainder of capturing unit 710.

Orientation adjustment module 602 may be configured to adjust anorientation of image sensor 220 of capturing unit 710. As discussedabove, image sensor 220 may be mounted on an orientation adjustment unit705 configured for movement. Orientation adjustment unit 705 may beconfigured for rotational and/or lateral movement in response tocommands from orientation adjustment module 602. In some embodimentsorientation adjustment unit 705 may be adjust an orientation of imagesensor 220 via motors, electromagnets, permanent magnets, and/or anysuitable combination thereof.

In some embodiments, monitoring module 603 may be provided forcontinuous monitoring. Such continuous monitoring may include tracking amovement of at least a portion of an object included in one or moreimages captured by the image sensor. For example, in one embodiment,apparatus 110 may track an object as long as the object remainssubstantially within the field-of-view of image sensor 220. Inadditional embodiments, monitoring module 603 may engage orientationadjustment module 602 to instruct orientation adjustment unit 705 tocontinually orient image sensor 220 towards an object of interest. Forexample, in one embodiment, monitoring module 603 may cause image sensor220 to adjust an orientation to ensure that a certain designated object,for example, the face of a particular person, remains within thefield-of view of image sensor 220, even as that designated object movesabout. In another embodiment, monitoring module 603 may continuouslymonitor an area of interest included in one or more images captured bythe image sensor. For example, a user may be occupied by a certain task,for example, typing on a laptop, while image sensor 220 remains orientedin a particular direction and continuously monitors a portion of eachimage from a series of images to detect a trigger or other event. Forexample, image sensor 210 may be oriented towards a piece of laboratoryequipment and monitoring module 603 may be configured to monitor astatus light on the laboratory equipment for a change in status, whilethe user's attention is otherwise occupied.

In some embodiments consistent with the present disclosure, capturingunit 710 may include a plurality of image sensors 220. The plurality ofimage sensors 220 may each be configured to capture different imagedata. For example, when a plurality of image sensors 220 are provided,the image sensors 220 may capture images having different resolutions,may capture wider or narrower fields of view, and may have differentlevels of magnification. Image sensors 220 may be provided with varyinglenses to permit these different configurations. In some embodiments, aplurality of image sensors 220 may include image sensors 220 havingdifferent orientations. Thus, each of the plurality of image sensors 220may be pointed in a different direction to capture different images. Thefields of view of image sensors 220 may be overlapping in someembodiments. The plurality of image sensors 220 may each be configuredfor orientation adjustment, for example, by being paired with an imageadjustment unit 705. In some embodiments, monitoring module 603, oranother module associated with memory 550, may be configured toindividually adjust the orientations of the plurality of image sensors220 as well as to turn each of the plurality of image sensors 220 on oroff as may be required. In some embodiments, monitoring an object orperson captured by an image sensor 220 may include tracking movement ofthe object across the fields of view of the plurality of image sensors220.

Embodiments consistent with the present disclosure may includeconnectors configured to connect a capturing unit and a power unit of awearable apparatus. Capturing units consistent with the presentdisclosure may include least one image sensor configured to captureimages of an environment of a user. Power units consistent with thepresent disclosure may be configured to house a power source and/or atleast one processing device. Connectors consistent with the presentdisclosure may be configured to connect the capturing unit and the powerunit, and may be configured to secure the apparatus to an article ofclothing such that the capturing unit is positioned over an outersurface of the article of clothing and the power unit is positionedunder an inner surface of the article of clothing. Exemplary embodimentsof capturing units, connectors, and power units consistent with thedisclosure are discussed in further detail with respect to FIGS. 8-14.

FIG. 8 is a schematic illustration of an embodiment of wearableapparatus 110 securable to an article of clothing consistent with thepresent disclosure. As illustrated in FIG. 8, capturing unit 710 andpower unit 720 may be connected by a connector 730 such that capturingunit 710 is positioned on one side of an article of clothing 750 andpower unit 720 is positioned on the opposite side of the clothing 750.In some embodiments, capturing unit 710 may be positioned over an outersurface of the article of clothing 750 and power unit 720 may be locatedunder an inner surface of the article of clothing 750. The power unit720 may be configured to be placed against the skin of a user.

Capturing unit 710 may include an image sensor 220 and an orientationadjustment unit 705 (as illustrated in FIG. 7). Power unit 720 mayinclude mobile power source 520 and processor 210. Power unit 720 mayfurther include any combination of elements previously discussed thatmay be a part of wearable apparatus 110, including, but not limited to,wireless transceiver 530, feedback outputting unit 230, memory 550, anddata port 570.

Connector 730 may include a clip 715 or other mechanical connectiondesigned to clip or attach capturing unit 710 and power unit 720 to anarticle of clothing 750 as illustrated in FIG. 8. As illustrated, clip715 may connect to each of capturing unit 710 and power unit 720 at aperimeter thereof, and may wrap around an edge of the article ofclothing 750 to affix the capturing unit 710 and power unit 720 inplace. Connector 730 may further include a power cable 760 and a datacable 770. Power cable 760 may be capable of conveying power from mobilepower source 520 to image sensor 220 of capturing unit 710. Power cable760 may also be configured to provide power to any other elements ofcapturing unit 710, e.g., orientation adjustment unit 705. Data cable770 may be capable of conveying captured image data from image sensor220 in capturing unit 710 to processor 800 in the power unit 720. Datacable 770 may be further capable of conveying additional data betweencapturing unit 710 and processor 800, e.g., control instructions fororientation adjustment unit 705.

FIG. 9 is a schematic illustration of a user 100 wearing a wearableapparatus 110 consistent with an embodiment of the present disclosure.As illustrated in FIG. 9, capturing unit 710 is located on an exteriorsurface of the clothing 750 of user 100. Capturing unit 710 is connectedto power unit 720 (not seen in this illustration) via connector 730,which wraps around an edge of clothing 750.

In some embodiments, connector 730 may include a flexible printedcircuit board (PCB). FIG. 10 illustrates an exemplary embodiment whereinconnector 730 includes a flexible printed circuit board 765. Flexibleprinted circuit board 765 may include data connections and powerconnections between capturing unit 710 and power unit 720. Thus, in someembodiments, flexible printed circuit board 765 may serve to replacepower cable 760 and data cable 770. In alternative embodiments, flexibleprinted circuit board 765 may be included in addition to at least one ofpower cable 760 and data cable 770. In various embodiments discussedherein, flexible printed circuit board 765 may be substituted for, orincluded in addition to, power cable 760 and data cable 770.

FIG. 11 is a schematic illustration of another embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure. As illustrated in FIG. 11, connector 730 may becentrally located with respect to capturing unit 710 and power unit 720.Central location of connector 730 may facilitate affixing apparatus 110to clothing 750 through a hole in clothing 750 such as, for example, abutton-hole in an existing article of clothing 750 or a specialty holein an article of clothing 750 designed to accommodate wearable apparatus110.

FIG. 12 is a schematic illustration of still another embodiment ofwearable apparatus 110 securable to an article of clothing. Asillustrated in FIG. 12, connector 730 may include a first magnet 731 anda second magnet 732. First magnet 731 and second magnet 732 may securecapturing unit 710 to power unit 720 with the article of clothingpositioned between first magnet 731 and second magnet 732. Inembodiments including first magnet 731 and second magnet 732, powercable 760 and data cable 770 may also be included. In these embodiments,power cable 760 and data cable 770 may be of any length, and may providea flexible power and data connection between capturing unit 710 andpower unit 720. Embodiments including first magnet 731 and second magnet732 may further include a flexible PCB 765 connection in addition to orinstead of power cable 760 and/or data cable 770. In some embodiments,first magnet 731 or second magnet 732 may be replaced by an objectcomprising a metal material.

FIG. 13 is a schematic illustration of yet another embodiment of awearable apparatus 110 securable to an article of clothing. FIG. 13illustrates an embodiment wherein power and data may be wirelesslytransferred between capturing unit 710 and power unit 720. Asillustrated in FIG. 13, first magnet 731 and second magnet 732 may beprovided as connector 730 to secure capturing unit 710 and power unit720 to an article of clothing 750. Power and/or data may be transferredbetween capturing unit 710 and power unit 720 via any suitable wirelesstechnology, for example, magnetic and/or capacitive coupling, near fieldcommunication technologies, radiofrequency transfer, and any otherwireless technology suitable for transferring data and/or power acrossshort distances.

FIG. 14 illustrates still another embodiment of wearable apparatus 110securable to an article of clothing 750 of a user. As illustrated inFIG. 14, connector 730 may include features designed for a contact fit.For example, capturing unit 710 may include a ring 733 with a hollowcenter having a diameter slightly larger than a disk-shaped protrusion734 located on power unit 720. When pressed together with fabric of anarticle of clothing 750 between them, disk-shaped protrusion 734 may fittightly inside ring 733, securing capturing unit 710 to power unit 720.FIG. 14 illustrates an embodiment that does not include any cabling orother physical connection between capturing unit 710 and power unit 720.In this embodiment, capturing unit 710 and power unit 720 may transferpower and data wirelessly. In alternative embodiments, capturing unit710 and power unit 720 may transfer power and data via at least one ofcable 760, data cable 770, and flexible printed circuit board 765.

FIG. 15 illustrates another aspect of power unit 720 consistent withembodiments described herein. Power unit 720 may be configured to bepositioned directly against the user's skin. To facilitate suchpositioning, power unit 720 may further include at least one surfacecoated with a biocompatible material 740. Biocompatible materials 740may include materials that will not negatively react with the skin ofthe user when worn against the skin for extended periods of time. Suchmaterials may include, for example, silicone, PTFE, kapton, polyimide,titanium, nitinol, platinum, and others. Also as illustrated in FIG. 15,power unit 720 may be sized such that an inner volume of the power unitis substantially filled by mobile power source 520. That is, in someembodiments, the inner volume of power unit 720 may be such that thevolume does not accommodate any additional components except for mobilepower source 520. In some embodiments, mobile power source 520 may takeadvantage of its close proximity to the skin of user's skin. Forexample, mobile power source 520 may use the Peltier effect to producepower and/or charge the power source.

In further embodiments, an apparatus securable to an article of clothingmay further include protective circuitry associated with power source520 housed in in power unit 720. FIG. 16 illustrates an exemplaryembodiment including protective circuitry 775. As illustrated in FIG.16, protective circuitry 775 may be located remotely with respect topower unit 720. In alternative embodiments, protective circuitry 775 mayalso be located in capturing unit 710, on flexible printed circuit board765, or in power unit 720.

Protective circuitry 775 may be configured to protect image sensor 220and/or other elements of capturing unit 710 from potentially dangerouscurrents and/or voltages produced by mobile power source 520. Protectivecircuitry 775 may include passive components such as capacitors,resistors, diodes, inductors, etc., to provide protection to elements ofcapturing unit 710. In some embodiments, protective circuitry 775 mayalso include active components, such as transistors, to provideprotection to elements of capturing unit 710. For example, in someembodiments, protective circuitry 775 may comprise one or more resistorsserving as fuses. Each fuse may comprise a wire or strip that melts(thereby braking a connection between circuitry of image capturing unit710 and circuitry of power unit 720) when current flowing through thefuse exceeds a predetermined limit (e.g., 500 milliamps, 900 milliamps,1 amp, 1.1 amps, 2 amp, 2.1 amps, 3 amps, etc.) Any or all of thepreviously described embodiments may incorporate protective circuitry775.

In some embodiments, the wearable apparatus may transmit data to acomputing device (e.g., a smartphone, tablet, watch, computer, etc.)over one or more networks via any known wireless standard (e.g.,cellular, Wi-Fi, Bluetooth®, etc.), or via near-filed capacitivecoupling, other short range wireless techniques, or via a wiredconnection. Similarly, the wearable apparatus may receive data from thecomputing device over one or more networks via any known wirelessstandard (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or via near-filedcapacitive coupling, other short range wireless techniques, or via awired connection. The data transmitted to the wearable apparatus and/orreceived by the wireless apparatus may include images, portions ofimages, identifiers related to information appearing in analyzed imagesor associated with analyzed audio, or any other data representing imageand/or audio data. For example, an image may be analyzed and anidentifier related to an activity occurring in the image may betransmitted to the computing device (e.g., the “paired device”). In theembodiments described herein, the wearable apparatus may process imagesand/or audio locally (on board the wearable apparatus) and/or remotely(via a computing device). Further, in the embodiments described herein,the wearable apparatus may transmit data related to the analysis ofimages and/or audio to a computing device for further analysis, display,and/or transmission to another device (e.g., a paired device). Further,a paired device may execute one or more applications (apps) to process,display, and/or analyze data (e.g., identifiers, text, images, audio,etc.) received from the wearable apparatus.

Some of the disclosed embodiments may involve systems, devices, methods,and software products for determining at least one keyword. For example,at least one keyword may be determined based on data collected byapparatus 110. At least one search query may be determined based on theat least one keyword. The at least one search query may be transmittedto a search engine.

In some embodiments, at least one keyword may be determined based on atleast one or more images captured by image sensor 220. In some cases,the at least one keyword may be selected from a keywords pool stored inmemory. In some cases, optical character recognition (OCR) may beperformed on at least one image captured by image sensor 220, and the atleast one keyword may be determined based on the OCR result. In somecases, at least one image captured by image sensor 220 may be analyzedto recognize: a person, an object, a location, a scene, and so forth.Further, the at least one keyword may be determined based on therecognized person, object, location, scene, etc. For example, the atleast one keyword may comprise: a person's name, an object's name, aplace's name, a date, a sport team's name, a movie's name, a book'sname, and so forth.

In some embodiments, at least one keyword may be determined based on theuser's behavior. The user's behavior may be determined based on ananalysis of the one or more images captured by image sensor 220. In someembodiments, at least one keyword may be determined based on activitiesof a user and/or other person. The one or more images captured by imagesensor 220 may be analyzed to identify the activities of the user and/orthe other person who appears in one or more images captured by imagesensor 220. In some embodiments, at least one keyword may be determinedbased on at least one or more audio segments captured by apparatus 110.In some embodiments, at least one keyword may be determined based on atleast GPS information associated with the user. In some embodiments, atleast one keyword may be determined based on at least the current timeand/or date.

In some embodiments, at least one search query may be determined basedon at least one keyword. In some cases, the at least one search querymay comprise the at least one keyword. In some cases, the at least onesearch query may comprise the at least one keyword and additionalkeywords provided by the user. In some cases, the at least one searchquery may comprise the at least one keyword and one or more images, suchas images captured by image sensor 220. In some cases, the at least onesearch query may comprise the at least one keyword and one or more audiosegments, such as audio segments captured by apparatus 110.

In some embodiments, the at least one search query may be transmitted toa search engine. In some embodiments, search results provided by thesearch engine in response to the at least one search query may beprovided to the user. In some embodiments, the at least one search querymay be used to access a database.

For example, in one embodiment, the keywords may include a name of atype of food, such as quinoa, or a brand name of a food product; and thesearch will output information related to desirable quantities ofconsumption, facts about the nutritional profile, and so forth. Inanother example, in one embodiment, the keywords may include a name of arestaurant, and the search will output information related to therestaurant, such as a menu, opening hours, reviews, and so forth. Thename of the restaurant may be obtained using OCR on an image of signage,using GPS information, and so forth. In another example, in oneembodiment, the keywords may include a name of a person, and the searchwill provide information from a social network profile of the person.The name of the person may be obtained using OCR on an image of a nametag attached to the person's shirt, using face recognition algorithms,and so forth. In another example, in one embodiment, the keywords mayinclude a name of a book, and the search will output information relatedto the book, such as reviews, sales statistics, information regardingthe author of the book, and so forth. In another example, in oneembodiment, the keywords may include a name of a movie, and the searchwill output information related to the movie, such as reviews, boxoffice statistics, information regarding the cast of the movie, showtimes, and so forth. In another example, in one embodiment, the keywordsmay include a name of a sport team, and the search will outputinformation related to the sport team, such as statistics, latestresults, future schedule, information regarding the players of the sportteam, and so forth. For example, the name of the sport team may beobtained using audio recognition algorithms.

Camera-Based Directional Hearing Aid

As discussed previously, the disclosed embodiments may include providingfeedback, such as acoustical and tactile feedback, to one or moreauxiliary devices in response to processing at least one image in anenvironment. In some embodiments, the auxiliary device may be anearpiece or other device used to provide auditory feedback to the user,such as a hearing aid. Traditional hearing aids often use microphones toamplify sounds in the user's environment. These traditional systems,however, are often unable to distinguish between sounds that may be ofparticular importance to the wearer of the device, or may do so on alimited basis. Using the systems and methods of the disclosedembodiments, various improvements to traditional hearing aids areprovided, as described in detail below.

In one embodiment, a camera-based directional hearing aid may beprovided for selectively amplifying sounds based on a look direction ofa user. The hearing aid may communicate with an image capturing device,such as apparatus 110, to determine the look direction of the user. Thislook direction may be used to isolate and/or selectively amplify soundsreceived from that direction (e.g., sounds from individuals in theuser's look direction, etc.). Sounds received from directions other thanthe user's look direction may be suppressed, attenuated, filtered or thelike.

FIG. 17A is a schematic illustration of an example of a user 100 wearingan apparatus 110 for a camera-based hearing interface device 1710according to a disclosed embodiment. User 100 may wear apparatus 110that is physically connected to a shirt or other piece of clothing ofuser 100, as shown. Consistent with the disclosed embodiments, apparatus110 may be positioned in other locations, as described previously. Forexample, apparatus 110 may be physically connected to a necklace, abelt, glasses, a wrist strap, a button, etc. Apparatus 110 may beconfigured to communicate with a hearing interface device such ashearing interface device 1710. Such communication may be through a wiredconnection, or may be made wirelessly (e.g., using a Bluetooth™, NFC, orforms of wireless communication). In some embodiments, one or moreadditional devices may also be included, such as computing device 120.Accordingly, one or more of the processes or functions described hereinwith respect to apparatus 110 or processor 210 may be performed bycomputing device 120 and/or processor 540.

Hearing interface device 1710 may be any device configured to provideaudible feedback to user 100. Hearing interface device 1710 maycorrespond to feedback outputting unit 230, described above, andtherefore any descriptions of feedback outputting unit 230 may alsoapply to hearing interface device 1710. In some embodiments, hearinginterface device 1710 may be separate from feedback outputting unit 230and may be configured to receive signals from feedback outputting unit230. As shown in FIG. 17A, hearing interface device 1710 may be placedin one or both ears of user 100, similar to traditional hearinginterface devices. Hearing interface device 1710 may be of variousstyles, including in-the-canal, completely-in-canal, in-the-ear,behind-the-ear, on-the-ear, receiver-in-canal, open fit, or variousother styles. Hearing interface device 1710 may include one or morespeakers for providing audible feedback to user 100, microphones fordetecting sounds in the environment of user 100, internal electronics,processors, memories, etc. In some embodiments, in addition to orinstead of a microphone, hearing interface device 1710 may comprise oneor more communication units, and in particular one or more receivers forreceiving signals from apparatus 110 and transferring the signals touser 100.

Hearing interface device 1710 may have various other configurations orplacement locations. In some embodiments, hearing interface device 1710may comprise a bone conduction headphone 1711, as shown in FIG. 17A.Bone conduction headphone 1711 may be surgically implanted and mayprovide audible feedback to user 100 through bone conduction of soundvibrations to the inner ear. Hearing interface device 1710 may alsocomprise one or more headphones (e.g., wireless headphones, over-earheadphones, etc.) or a portable speaker carried or worn by user 100. Insome embodiments, hearing interface device 1710 may be integrated intoother devices, such as a Bluetooth™ headset of the user, glasses, ahelmet (e.g., motorcycle helmets, bicycle helmets, etc.), a hat, etc.

Apparatus 110 may be configured to determine a user look direction 1750of user 100. In some embodiments, user look direction 1750 may betracked by monitoring a direction of the chin, or another body part orface part of user 100 relative to an optical axis of a camera sensor1751. Apparatus 110 may be configured to capture one or more images ofthe surrounding environment of user, for example, using image sensor220. The captured images may include a representation of a chin of user100, which may be used to determine user look direction 1750. Processor210 (and/or processors 210 a and 210 b) may be configured to analyze thecaptured images and detect the chin or another part of user 100 usingvarious image detection or processing algorithms (e.g., usingconvolutional neural networks (CNN), scale-invariant feature transform(SIFT), histogram of oriented gradients (HOG) features, or othertechniques). Based on the detected representation of a chin of user 100,look direction 1750 may be determined. Look direction 1750 may bedetermined in part by comparing the detected representation of a chin ofuser 100 to an optical axis of a camera sensor 1751. For example, theoptical axis 1751 may be known or fixed in each image and processor 210may determine look direction 1750 by comparing a representative angle ofthe chin of user 100 to the direction of optical axis 1751. While theprocess is described using a representation of a chin of user 100,various other features may be detected for determining user lookdirection 1750, including the user's face, nose, eyes, hand, etc.

In other embodiments, user look direction 1750 may be aligned moreclosely with the optical axis 1751. For example, as discussed above,apparatus 110 may be affixed to a pair of glasses of user 100, as shownin FIG. 1A. In this embodiment, user look direction 1750 may be the sameas or close to the direction of optical axis 1751. Accordingly, userlook direction 1750 may be determined or approximated based on the viewof image sensor 220.

FIG. 17B is a schematic illustration of an embodiment of an apparatussecurable to an article of clothing consistent with the presentdisclosure. Apparatus 110 may be securable to a piece of clothing, suchas the shirt of user 110, as shown in FIG. 17A. Apparatus 110 may besecurable to other articles of clothing, such as a belt or pants of user100, as discussed above. Apparatus 110 may have one or more cameras1730, which may correspond to image sensor 220. Camera 1730 may beconfigured to capture images of the surrounding environment of user 100.In some embodiments, camera 1730 may be configured to detect arepresentation of a chin of the user in the same images capturing thesurrounding environment of the user, which may be used for otherfunctions described in this disclosure. In other embodiments camera 1730may be an auxiliary or separate camera dedicated to determining userlook direction 1750.

Apparatus 110 may further comprise one or more microphones 1720 forcapturing sounds from the environment of user 100. Microphone 1720 mayalso be configured to determine a directionality of sounds in theenvironment of user 100. For example, microphone 1720 may comprise oneor more directional microphones, which may be more sensitive to pickingup sounds in certain directions. For example, microphone 1720 maycomprise a unidirectional microphone, designed to pick up sound from asingle direction or small range of directions. Microphone 1720 may alsocomprise a cardioid microphone, which may be sensitive to sounds fromthe front and sides. Microphone 1720 may also include a microphonearray, which may comprise additional microphones, such as microphone1721 on the front of apparatus 110, or microphone 1722, placed on theside of apparatus 110. In some embodiments, microphone 1720 may be amulti-port microphone for capturing multiple audio signals. Themicrophones shown in FIG. 17B are by way of example only, and anysuitable number, configuration, or location of microphones may beutilized. Processor 210 may be configured to distinguish sounds withinthe environment of user 100 and determine an approximate directionalityof each sound. For example, using an array of microphones 1720,processor 210 may compare the relative timing or amplitude of anindividual sound among the microphones 1720 to determine adirectionality relative to apparatus 100.

As a preliminary step before other audio analysis operations, the soundcaptured from an environment of a user may be classified using any audioclassification technique. For example, the sound may be classified intosegments containing music, tones, laughter, screams, or the like.Indications of the respective segments may be logged in a database andmay prove highly useful for life logging applications. As one example,the logged information may enable the system to to retrieve and/ordetermine a mood when the user met another person. Additionally, suchprocessing is relatively fast and efficient, and does not requiresignificant computing resources, and transmitting the information to adestination does not require significant bandwidth. Moreover, oncecertain parts of the audio are classified as non-speech, more computingresources may be available for processing the other segments.

Based on the determined user look direction 1750, processor 210 mayselectively condition or amplify sounds from a region associated withuser look direction 1750. FIG. 18 is a schematic illustration showing anexemplary environment for use of a camera-based hearing aid consistentwith the present disclosure. Microphone 1720 may detect one or moresounds 1820, 1821, and 1822 within the environment of user 100. Based onuser look direction 1750, determined by processor 210, a region 1830associated with user look direction 1750 may be determined. As shown inFIG. 18, region 1830 may be defined by a cone or range of directionsbased on user look direction 1750. The range of angles may be defined byan angle, θ, as shown in FIG. 18. The angle, θ, may be any suitableangle for defining a range for conditioning sounds within theenvironment of user 100 (e.g., 10 degrees, 20 degrees, 45 degrees).

Processor 210 may be configured to cause selective conditioning ofsounds in the environment of user 100 based on region 1830. Theconditioned audio signal may be transmitted to hearing interface device1710, and thus may provide user 100 with audible feedback correspondingto the look direction of the user. For example, processor 210 maydetermine that sound 1820 (which may correspond to the voice of anindividual 1810, or to noise for example) is within region 1830.Processor 210 may then perform various conditioning techniques on theaudio signals received from microphone 1720. The conditioning mayinclude amplifying audio signals determined to correspond to sound 1820relative to other audio signals. Amplification may be accomplisheddigitally, for example by processing audio signals associated with 1820relative to other signals. Amplification may also be accomplished bychanging one or more parameters of microphone 1720 to focus on audiosounds emanating from region 1830 (e.g., a region of interest)associated with user look direction 1750. For example, microphone 1720may be a directional microphone that and processor 210 may perform anoperation to focus microphone 1720 on sound 1820 or other sounds withinregion 1830. Various other techniques for amplifying sound 1820 may beused, such as using a beamforming microphone array, acoustic telescopetechniques, etc.

Conditioning may also include attenuation or suppressing one or moreaudio signals received from directions outside of region 1830. Forexample, processor 1820 may attenuate sounds 1821 and 1822. Similar toamplification of sound 1820, attenuation of sounds may occur throughprocessing audio signals, or by varying one or more parametersassociated with one or more microphones 1720 to direct focus away fromsounds emanating from outside of region 1830.

In some embodiments, conditioning may further include changing a tone ofaudio signals corresponding to sound 1820 to make sound 1820 moreperceptible to user 100. For example, user 100 may have lessersensitivity to tones in a certain range and conditioning of the audiosignals may adjust the pitch of sound 1820 to make it more perceptibleto user 100. For example, user 100 may experience hearing loss infrequencies above 10 khz. Accordingly, processor 210 may remap higherfrequencies (e.g., at 15 khz) to 10 khz. In some embodiments processor210 may be configured to change a rate of speech associated with one ormore audio signals. Accordingly, processor 210 may be configured todetect speech within one or more audio signals received by microphone1720, for example using voice activity detection (VAD) algorithms ortechniques. If sound 1820 is determined to correspond to voice orspeech, for example from individual 1810, processor 220 may beconfigured to vary the playback rate of sound 1820. For example, therate of speech of individual 1810 may be decreased to make the detectedspeech more perceptible to user 100. Various other processing may beperformed, such as modifying the tone of sound 1820 to maintain the samepitch as the original audio signal, or to reduce noise within the audiosignal. If speech recognition has been performed on the audio signalassociated with sound 1820, conditioning may further include modifyingthe audio signal based on the detected speech. For example, processor210 may introduce pauses or increase the duration of pauses betweenwords and/or sentences, which may make the speech easier to understand.

The conditioned audio signal may then be transmitted to hearinginterface device 1710 and produced for user 100. Thus, in theconditioned audio signal, sound 1820 may be easier to hear to user 100,louder and/or more easily distinguishable than sounds 1821 and 1822,which may represent background noise within the environment.

FIG. 19 is a flowchart showing an exemplary process 1900 for selectivelyamplifying sounds emanating from a detected look direction of a userconsistent with disclosed embodiments. Process 1900 may be performed byone or more processors associated with apparatus 110, such as processor210. In some embodiments, some or all of process 1900 may be performedon processors external to apparatus 110. In other words, the processorperforming process 1900 may be included in a common housing asmicrophone 1720 and camera 1730, or may be included in a second housing.For example, one or more portions of process 1900 may be performed byprocessors in hearing interface device 1710, or an auxiliary device,such as computing device 120.

In step 1910, process 1900 may include receiving a plurality of imagesfrom an environment of a user captured by a camera. The camera may be awearable camera such as camera 1730 of apparatus 110. In step 1912,process 1900 may include receiving audio signals representative ofsounds received by at least one microphone. The microphone may beconfigured to capture sounds from an environment of the user. Forexample, the microphone may be microphone 1720, as described above.Accordingly, the microphone may include a directional microphone, amicrophone array, a multi-port microphone, or various other types ofmicrophones. In some embodiments, the microphone and wearable camera maybe included in a common housing, such as the housing of apparatus 110.The one or more processors performing process 1900 may also be includedin the housing or may be included in a second housing. In suchembodiments, the processor(s) may be configured to receive images and/oraudio signals from the common housing via a wireless link (e.g.,Bluetooth™, NFC, etc.). Accordingly, the common housing (e.g., apparatus110) and the second housing (e.g., computing device 120) may furthercomprise transmitters or various other communication components.

In step 1914, process 1900 may include determining a look direction forthe user based on analysis of at least one of the plurality of images.As discussed above, various techniques may be used to determine the userlook direction. In some embodiments, the look direction may bedetermined based, at least in part, upon detection of a representationof a chin of a user in one or more images. The images may be processedto determine a pointing direction of the chin relative to an opticalaxis of the wearable camera, as discussed above.

In step 1916, process 1900 may include causing selective conditioning ofat least one audio signal received by the at least one microphone from aregion associated with the look direction of the user. As describedabove, the region may be determined based on the user look directiondetermined in step 1914. The range may be associated with an angularwidth about the look direction (e.g., 10 degrees, 20 degrees, 45degrees, etc.). Various forms of conditioning may be performed on theaudio signal, as discussed above. In some embodiments, conditioning mayinclude changing the tone or playback speed of an audio signal. Forexample, conditioning may include changing a rate of speech associatedwith the audio signal. In some embodiments, the conditioning may includeamplification of the audio signal relative to other audio signalsreceived from outside of the region associated with the look directionof the user. Amplification may be performed by various means, such asoperation of a directional microphone configured to focus on audiosounds emanating from the region, or varying one or more parametersassociated with the microphone to cause the microphone to focus on audiosounds emanating from the region. The amplification may includeattenuating or suppressing one or more audio signals received by themicrophone from directions outside the region associated with the lookdirection of user 110.

In step 1918, process 1900 may include causing transmission of the atleast one conditioned audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user. The conditioned audiosignal, for example, may be transmitted to hearing interface device1710, which may provide sound corresponding to the audio signal to user100. The processor performing process 1900 may further be configured tocause transmission to the hearing interface device of one or more audiosignals representative of background noise, which may be attenuatedrelative to the at least one conditioned audio signal. For example,processor 220 may be configured to transmit audio signals correspondingto sounds 1820, 1821, and 1822. The signal associated with 1820,however, may be modified in a different manner, for example amplified,from sounds 1821 and 1822 based on a determination that sound 1820 iswithin region 1830. In some embodiments, hearing interface device 1710may include a speaker associated with an earpiece. For example, hearinginterface device may be inserted at least partially into the ear of theuser for providing audio to the user. Hearing interface device may alsobe external to the ear, such as a behind-the-ear hearing device, one ormore headphones, a small portable speaker, or the like. In someembodiments, hearing interface device may include a bone conductionmicrophone, configured to provide an audio signal to user throughvibrations of a bone of the user's head. Such devices may be placed incontact with the exterior of the user's skin, or may be implantedsurgically and attached to the bone of the user.

Hearing Aid with Voice and/or Image Recognition

Consistent with the disclosed embodiments, a hearing aid may selectivelyamplify audio signals associated with a voice of a recognizedindividual. The hearing aid system may store voice characteristicsand/or facial features of a recognized person to aid in recognition andselective amplification. For example, when an individual enters thefield of view of apparatus 110, the individual may be recognized as anindividual that has been introduced to the device, or that has possiblyinteracted with user 100 in the past (e.g., a friend, colleague,relative, prior acquaintance, etc.). Accordingly, audio signalsassociated with the recognized individual's voice may be isolated and/orselectively amplified relative to other sounds in the environment of theuser. Audio signals associated with sounds received from directionsother than the individual's direction may be suppressed, attenuated,filtered or the like.

User 100 may wear a hearing aid device similar to the camera-basedhearing aid device discussed above. For example, the hearing aid devicemay be hearing interface device 1720, as shown in FIG. 17A. Hearinginterface device 1710 may be any device configured to provide audiblefeedback to user 100. Hearing interface device 1710 may be placed in oneor both ears of user 100, similar to traditional hearing interfacedevices. As discussed above, hearing interface device 1710 may be ofvarious styles, including in-the-canal, completely-in-canal, in-the-ear,behind-the-ear, on-the-ear, receiver-in-canal, open fit, or variousother styles. Hearing interface device 1710 may include one or morespeakers for providing audible feedback to user 100, a communicationunit for receiving signals from another system, such as apparatus 110,microphones for detecting sounds in the environment of user 100,internal electronics, processors, memories, etc. Hearing interfacedevice 1710 may correspond to feedback outputting unit 230 or may beseparate from feedback outputting unit 230 and may be configured toreceive signals from feedback outputting unit 230.

In some embodiments, hearing interface device 1710 may comprise a boneconduction headphone 1711, as shown in FIG. 17A. Bone conductionheadphone 1711 may be surgically implanted and may provide audiblefeedback to user 100 through bone conduction of sound vibrations to theinner ear. Hearing interface device 1710 may also comprise one or moreheadphones (e.g., wireless headphones, over-ear headphones, etc.) or aportable speaker carried or worn by user 100. In some embodiments,hearing interface device 1710 may be integrated into other devices, suchas a Bluetooth™ headset of the user, glasses, a helmet (e.g., motorcyclehelmets, bicycle helmets, etc.), a hat, etc.

Hearing interface device 1710 may be configured to communicate with acamera device, such as apparatus 110. Such communication may be througha wired connection, or may be made wirelessly (e.g., using a Bluetooth™,NFC, or forms of wireless communication). As discussed above, apparatus110 may be worn by user 100 in various configurations, including beingphysically connected to a shirt, necklace, a belt, glasses, a wriststrap, a button, or other articles associated with user 100. In someembodiments, one or more additional devices may also be included, suchas computing device 120. Accordingly, one or more of the processes orfunctions described herein with respect to apparatus 110 or processor210 may be performed by computing device 120 and/or processor 540.

As discussed above, apparatus 110 may comprise at least one microphoneand at least one image capture device. Apparatus 110 may comprisemicrophone 1720, as described with respect to FIG. 17B. Microphone 1720may be configured to determine a directionality of sounds in theenvironment of user 100. For example, microphone 1720 may comprise oneor more directional microphones, a microphone array, a multi-portmicrophone, or the like. The microphones shown in FIG. 17B are by way ofexample only, and any suitable number, configuration, or location ofmicrophones may be utilized. Processor 210 may be configured todistinguish sounds within the environment of user 100 and determine anapproximate directionality of each sound. For example, using an array ofmicrophones 1720, processor 210 may compare the relative timing oramplitude of an individual sound among the microphones 1720 to determinea directionality relative to apparatus 100. Apparatus 110 may compriseone or more cameras, such as camera 1730, which may correspond to imagesensor 220. Camera 1730 may be configured to capture images of thesurrounding environment of user 100.

Apparatus 110 may be configured to recognize an individual in theenvironment of user 100. FIG. 20A is a schematic illustration showing anexemplary environment for use of a hearing aid with voice and/or imagerecognition consistent with the present disclosure. Apparatus 110 may beconfigured to recognize a face 2011 or voice 2012 associated with anindividual 2010 within the environment of user 100. For example,apparatus 110 may be configured to capture one or more images of thesurrounding environment of user 100 using camera 1730. The capturedimages may include a representation of a recognized individual 2010,which may be a friend, colleague, relative, or prior acquaintance ofuser 100. Processor 210 (and/or processors 210 a and 210 b) may beconfigured to analyze the captured images and detect the recognized userusing various facial recognition techniques, as represented by element2011. Accordingly, apparatus 110, or specifically memory 550, maycomprise one or more facial or voice recognition components.

FIG. 20B illustrates an exemplary embodiment of apparatus 110 comprisingfacial and voice recognition components consistent with the presentdisclosure. Apparatus 110 is shown in FIG. 20B in a simplified form, andapparatus 110 may contain additional elements or may have alternativeconfigurations, for example, as shown in FIGS. 5A-5C. Memory 550 (or 550a or 550 b) may include facial recognition component 2040 and voicerecognition component 2041. These components may be instead of or inaddition to orientation identification module 601, orientationadjustment module 602, and motion tracking module 603 as shown in FIG.6. Components 2040 and 2041 may contain software instructions forexecution by at least one processing device, e.g., processor 210,included with a wearable apparatus. Components 2040 and 2041 are shownwithin memory 550 by way of example only, and may be located in otherlocations within the system. For example, components 2040 and 2041 maybe located in hearing interface device 1710, in computing device 120, ona remote server, or in another associated device.

Facial recognition component 2040 may be configured to identify one ormore faces within the environment of user 100. For example, facialrecognition component 2040 may identify facial features on the face 2011of individual 2010, such as the eyes, nose, cheekbones, jaw, or otherfeatures. Facial recognition component 2040 may then analyze therelative size and position of these features to identify the user.Facial recognition component 2040 may utilize one or more algorithms foranalyzing the detected features, such as principal component analysis(e.g., using eigenfaces), linear discriminant analysis, elastic bunchgraph matching (e.g., using Fisherface), Local Binary PatternsHistograms (LBPH), Scale Invariant Feature Transform (SIFT), Speed UpRobust Features (SURF), or the like. Other facial recognition techniquessuch as 3-Dimensional recognition, skin texture analysis, and/or thermalimaging may also be used to identify individuals. Other features besidesfacial features may also be used for identification, such as the height,body shape, or other distinguishing features of individual 2010.

Facial recognition component 2040 may access a database or dataassociated with user 100 to determine if the detected facial featurescorrespond to a recognized individual. For example, a processor 210 mayaccess a database 2050 containing information about individuals known touser 100 and data representing associated facial features or otheridentifying features. Such data may include one or more images of theindividuals, or data representative of a face of the user that may beused for identification through facial recognition. Database 2050 may beany device capable of storing information about one or more individuals,and may include a hard drive, a solid state drive, a web storageplatform, a remote server, or the like. Database 2050 may be locatedwithin apparatus 110 (e.g., within memory 550) or external to apparatus110, as shown in FIG. 20B. In some embodiments, database 2050 may beassociated with a social network platform, such as Facebook™, LinkedIn™,Instagram™, etc. Facial recognition component 2040 may also access acontact list of user 100, such as a contact list on the user's phone, aweb-based contact list (e.g., through Outlook™, Skype™, Google™,SalesForce™, etc.) or a dedicated contact list associated with hearinginterface device 1710. In some embodiments, database 2050 may becompiled by apparatus 110 through previous facial recognition analysis.For example, processor 210 may be configured to store data associatedwith one or more faces recognized in images captured by apparatus 110 indatabase 2050. Each time a face is detected in the images, the detectedfacial features or other data may be compared to previously identifiedfaces in database 2050. Facial recognition component 2040 may determinethat an individual is a recognized individual of user 100 if theindividual has previously been recognized by the system in a number ofinstances exceeding a certain threshold, if the individual has beenexplicitly introduced to apparatus 110, or the like.

In some embodiments, user 100 may have access to database 2050, such asthrough a web interface, an application on a mobile device, or throughapparatus 110 or an associated device. For example, user 100 may be ableto select which contacts are recognizable by apparatus 110 and/or deleteor add certain contacts manually. In some embodiments, a user oradministrator may be able to train facial recognition component 2040.For example, user 100 may have an option to confirm or rejectidentifications made by facial recognition component 2040, which mayimprove the accuracy of the system. This training may occur in realtime, as individual 2010 is being recognized, or at some later time.

Other data or information may also inform the facial identificationprocess. In some embodiments, processor 210 may use various techniquesto recognize the voice of individual 2010, as described in furtherdetail below. The recognized voice pattern and the detected facialfeatures may be used, either alone or in combination, to determine thatindividual 2010 is recognized by apparatus 110. Processor 210 may alsodetermine a user look direction 1750, as described above, which may beused to verify the identity of individual 2010. For example, if user 100is looking in the direction of individual 2010 (especially for aprolonged period), this may indicate that individual 2010 is recognizedby user 100, which may be used to increase the confidence of facialrecognition component 2040 or other identification means.

Processor 210 may further be configured to determine whether individual2010 is recognized by user 100 based on one or more detected audiocharacteristics of sounds associated with a voice of individual 2010.Returning to FIG. 20A, processor 210 may determine that sound 2020corresponds to voice 2012 of user 2010. Processor 210 may analyze audiosignals representative of sound 2020 captured by microphone 1720 todetermine whether individual 2010 is recognized by user 100. This may beperformed using voice recognition component 2041 (FIG. 20B) and mayinclude one or more voice recognition algorithms, such as Hidden MarkovModels, Dynamic Time Warping, neural networks, or other techniques.Voice recognition component and/or processor 210 may access database2050, which may further include a voiceprint of one or more individuals.Voice recognition component 2041 may analyze the audio signalrepresentative of sound 2020 to determine whether voice 2012 matches avoiceprint of an individual in database 2050. Accordingly, database 2050may contain voiceprint data associated with a number of individuals,similar to the stored facial identification data described above. Afterdetermining a match, individual 2010 may be determined to be arecognized individual of user 100. This process may be used alone, or inconjunction with the facial recognition techniques described above. Forexample, individual 2010 may be recognized using facial recognitioncomponent 2040 and may be verified using voice recognition component2041, or vice versa.

In some embodiments, apparatus 110 may detect the voice of an individualthat is not within the field of view of apparatus 110. For example, thevoice may be heard over a speakerphone, from a back seat, or the like.In such embodiments, recognition of an individual may be based on thevoice of the individual only, in the absence of a speaker in the fieldof view. Processor 110 may analyze the voice of the individual asdescribed above, for example, by determining whether the detected voicematches a voiceprint of an individual in database 2050.

After determining that individual 2010 is a recognized individual ofuser 100, processor 210 may cause selective conditioning of audioassociated with the recognized individual. The conditioned audio signalmay be transmitted to hearing interface device 1710, and thus mayprovide user 100 with audio conditioned based on the recognizedindividual. For example, the conditioning may include amplifying audiosignals determined to correspond to sound 2020 (which may correspond tovoice 2012 of individual 2010) relative to other audio signals. In someembodiments, amplification may be accomplished digitally, for example byprocessing audio signals associated with sound 2020 relative to othersignals. Additionally, or alternatively, amplification may beaccomplished by changing one or more parameters of microphone 1720 tofocus on audio sounds associated with individual 2010. For example,microphone 1720 may be a directional microphone and processor 210 mayperform an operation to focus microphone 1720 on sound 2020. Variousother techniques for amplifying sound 2020 may be used, such as using abeamforming microphone array, acoustic telescope techniques, etc.

In some embodiments, selective conditioning may include attenuation orsuppressing one or more audio signals received from directions notassociated with individual 2010. For example, processor 210 mayattenuate sounds 2021 and/or 2022. Similar to amplification of sound2020, attenuation of sounds may occur through processing audio signals,or by varying one or more parameters associated with microphone 1720 todirect focus away from sounds not associated with individual 2010.

Selective conditioning may further include determining whetherindividual 2010 is speaking. For example, processor 210 may beconfigured to analyze images or videos containing representations ofindividual 2010 to determine when individual 2010 is speaking, forexample, based on detected movement of the recognized individual's lips.This may also be determined through analysis of audio signals receivedby microphone 1720, for example by detecting the voice 2012 ofindividual 2010. In some embodiments, the selective conditioning mayoccur dynamically (initiated and/or terminated) based on whether or notthe recognized individual is speaking.

In some embodiments, conditioning may further include changing a tone ofone or more audio signals corresponding to sound 2020 to make the soundmore perceptible to user 100. For example, user 100 may have lessersensitivity to tones in a certain range and conditioning of the audiosignals may adjust the pitch of sound 2020. In some embodimentsprocessor 210 may be configured to change a rate of speech associatedwith one or more audio signals. For example, sound 2020 may bedetermined to correspond to voice 2012 of individual 2010. Processor 210may be configured to vary the rate of speech of individual 2010 to makethe detected speech more perceptible to user 100. Various otherprocessing may be performed, such as modifying the tone of sound 2020 tomaintain the same pitch as the original audio signal, or to reduce noisewithin the audio signal.

In some embodiments, processor 210 may determine a region 2030associated with individual 2010. Region 2030 may be associated with adirection of individual 2010 relative to apparatus 110 or user 100. Thedirection of individual 2010 may be determined using camera 1730 and/ormicrophone 1720 using the methods described above. As shown in FIG. 20A,region 2030 may be defined by a cone or range of directions based on adetermined direction of individual 2010. The range of angles may bedefined by an angle, θ, as shown in FIG. 20A. The angle, θ, may be anysuitable angle for defining a range for conditioning sounds within theenvironment of user 100 (e.g., 10 degrees, 20 degrees, 45 degrees).Region 2030 may be dynamically calculated as the position of individual2010 changes relative to apparatus 110. For example, as user 100 turns,or if individual 1020 moves within the environment, processor 210 may beconfigured to track individual 2010 within the environment anddynamically update region 2030. Region 2030 may be used for selectiveconditioning, for example by amplifying sounds associated with region2030 and/or attenuating sounds determined to be emanating from outsideof region 2030.

The conditioned audio signal may then be transmitted to hearinginterface device 1710 and produced for user 100. Thus, in theconditioned audio signal, sound 2020 (and specifically voice 2012) maybe louder and/or more easily distinguishable than sounds 2021 and 2022,which may represent background noise within the environment.

In some embodiments, processor 210 may perform further analysis based oncaptured images or videos to determine how to selectively conditionaudio signals associated with a recognized individual. In someembodiments, processor 210 may analyze the captured images toselectively condition audio associated with one individual relative toothers. For example, processor 210 may determine the direction of arecognized individual relative to the user based on the images and maydetermine how to selectively condition audio signals associated with theindividual based on the direction. If the recognized individual isstanding to the front of the user, audio associated with that user maybe amplified (or otherwise selectively conditioned) relative to audioassociated with an individual standing to the side of the user.Similarly, processor 210 may selectively condition audio signalsassociated with an individual based on proximity to the user. Processor210 may determine a distance from the user to each individual based oncaptured images and may selectively condition audio signals associatedwith the individuals based on the distance. For example, an individualcloser to the user may be prioritized higher than an individual that isfarther away.

In some embodiments, selective conditioning of audio signals associatedwith a recognized individual may be based on the identities ofindividuals within the environment of the user. For example, wheremultiple individuals are detected in the images, processor 210 may useone or more facial recognition techniques to identify the individuals,as described above. Audio signals associated with individuals that areknown to user 100 may be selectively amplified or otherwise conditionedto have priority over unknown individuals. For example, processor 210may be configured to attenuate or silence audio signals associated withbystanders in the user's environment, such as a noisy office mate, etc.In some embodiments, processor 210 may also determine a hierarchy ofindividuals and give priority based on the relative status of theindividuals. This hierarchy may be based on the individual's positionwithin a family or an organization (e.g., a company, sports team, club,etc.) relative to the user. For example, the user's boss may be rankedhigher than a co-worker or a member of the maintenance staff and thusmay have priority in the selective conditioning process. In someembodiments, the hierarchy may be determined based on a list ordatabase. Individuals recognized by the system may be rankedindividually or grouped into tiers of priority. This database may bemaintained specifically for this purpose, or may be accessed externally.For example, the database may be associated with a social network of theuser (e.g., Facebook™, LinkedIn™, etc.) and individuals may beprioritized based on their grouping or relationship with the user.Individuals identified as “close friends” or family, for example, may beprioritized over acquaintances of the user.

Selective conditioning may be based on a determined behavior of one ormore individuals determined based on the captured images. In someembodiments, processor 210 may be configured to determine a lookdirection of the individuals in the images. Accordingly, the selectiveconditioning may be based on behavior of the other individuals towardsthe recognized individual. For example, processor 210 may selectivelycondition audio associated with a first individual that one or moreother users are looking at. If the attention of the individuals shiftsto a second individual, processor 210 may then switch to selectivelycondition audio associated with the second user. In some embodiments,processor 210 may be configured to selectively condition audio based onwhether a recognized individual is speaking to the user or to anotherindividual. For example, when the recognized individual is speaking tothe user, the selective conditioning may include amplifying an audiosignal associated with the recognized individual relative to other audiosignals received from directions outside a region associated with therecognized individual. When the recognized individual is speaking toanother individual, the selective conditioning may include attenuatingthe audio signal relative to other audio signals received fromdirections outside the region associated with the recognized individual.

In some embodiments, processor 210 may have access to one or morevoiceprints of individuals, which may facilitate selective conditioningof voice 2012 of individual 2010 in relation to other sounds or voices.Having a speaker's voiceprint, and a high quality voiceprint inparticular, may provide for fast and efficient speaker separation. Ahigh quality voice print may be collected, for example, when the userspeaks alone, preferably in a quiet environment. By having a voiceprintof one or more speakers, it is possible to separate an ongoing voicesignal almost in real time, e.g. with a minimal delay, using a slidingtime window. The delay may be, for example 10 ms, 20 ms, 30 ms, 50 ms,100 ms, or the like. Different time windows may be selected, dependingon the quality of the voice print, on the quality of the captured audio,the difference in characteristics between the speaker and otherspeaker(s), the available processing resources, the required separationquality, or the like. In some embodiments, a voice print may beextracted from a segment of a conversation in which an individual speaksalone, and then used for separating the individual's voice later in theconversation, whether the individual's is recognized or not.

Separating voices may be performed as follows: spectral features, alsoreferred to as spectral attributes, spectral envelope, or spectrogrammay be extracted from a clean audio of a single speaker and fed into apre-trained first neural network, which generates or updates a signatureof the speaker's voice based on the extracted features. The audio may befor example, of one second of clean voice. The output signature may be avector representing the speaker's voice, such that the distance betweenthe vector and another vector extracted from the voice of the samespeaker is typically smaller than the distance between the vector and avector extracted from the voice of another speaker. The speaker's modelmay be pre-generated from a captured audio. Alternatively oradditionally, the model may be generated after a segment of the audio inwhich only the speaker speaks, followed by another segment in which thespeaker and another speaker (or background noise) is heard, and which itis required to separate.

Then, to separate the speaker's voice from additional speakers orbackground noise in a noisy audio, a second pre-trained neural networkmay receive the noisy audio and the speaker's signature, and output anaudio (which may also be represented as attributes) of the voice of thespeaker as extracted from the noisy audio, separated from the otherspeech or background noise. It will be appreciated that the same oradditional neural networks may be used to separate the voices ofmultiple speakers. For example, if there are two possible speakers, twoneural networks may be activated, each with models of the same noisyoutput and one of the two speakers. Alternatively, a neural network mayreceive voice signatures of two or more speakers, and output the voiceof each of the speakers separately. Accordingly, the system may generatetwo or more different audio outputs, each comprising the speech of therespective speaker. In some embodiments, if separation is impossible,the input voice may only be cleaned from background noise.

FIG. 21 is a flowchart showing an exemplary process 2100 for selectivelyamplifying audio signals associated with a voice of a recognizedindividual consistent with disclosed embodiments. Process 2100 may beperformed by one or more processors associated with apparatus 110, suchas processor 210. In some embodiments, some or all of process 2100 maybe performed on processors external to apparatus 110. In other words,the processor performing process 2100 may be included in the same commonhousing as microphone 1720 and camera 1730, or may be included in asecond housing. For example, one or more portions of process 2100 may beperformed by processors in hearing interface device 1710, or in anauxiliary device, such as computing device 120.

In step 2110, process 2100 may include receiving a plurality of imagesfrom an environment of a user captured by a camera. The images may becaptured by a wearable camera such as camera 1730 of apparatus 110. Instep 2112, process 2100 may include identifying a representation of arecognized individual in at least one of the plurality of images.Individual 2010 may be recognized by processor 210 using facialrecognition component 2040, as described above. For example, individual2010 may be a friend, colleague, relative, or prior acquaintance of theuser. Processor 210 may determine whether an individual represented inat least one of the plurality of images is a recognized individual basedon one or more detected facial features associated with the individual.Processor 210 may also determine whether the individual is recognizedbased on one or more detected audio characteristics of sounds determinedto be associated with a voice of the individual, as described above.

In step 2114, process 2100 may include receiving audio signalsrepresentative of sounds captured by a microphone. For example,apparatus 110 may receive audio signals representative of sounds 2020,2021, and 2022, captured by microphone 1720. Accordingly, the microphonemay include a directional microphone, a microphone array, a multi-portmicrophone, or various other types of microphones, as described above.In some embodiments, the microphone and wearable camera may be includedin a common housing, such as the housing of apparatus 110. The one ormore processors performing process 2100 may also be included in thehousing (e.g., processor 210), or may be included in a second housing.Where a second housing is used, the processor(s) may be configured toreceive images and/or audio signals from the common housing via awireless link (e.g., Bluetooth™, NFC, etc.). Accordingly, the commonhousing (e.g., apparatus 110) and the second housing (e.g., computingdevice 120) may further comprise transmitters, receivers, and/or variousother communication components.

In step 2116, process 2100 may include cause selective conditioning ofat least one audio signal received by the at least one microphone from aregion associated with the at least one recognized individual. Asdescribed above, the region may be determined based on a determineddirection of the recognized individual based one or more of theplurality of images or audio signals. The range may be associated withan angular width about the direction of the recognized individual (e.g.,10 degrees, 20 degrees, 45 degrees, etc.).

Various forms of conditioning may be performed on the audio signal, asdiscussed above. In some embodiments, conditioning may include changingthe tone or playback speed of an audio signal. For example, conditioningmay include changing a rate of speech associated with the audio signal.In some embodiments, the conditioning may include amplification of theaudio signal relative to other audio signals received from outside ofthe region associated with the recognized individual. Amplification maybe performed by various means, such as operation of a directionalmicrophone configured to focus on audio sounds emanating from the regionor varying one or more parameters associated with the microphone tocause the microphone to focus on audio sounds emanating from the region.The amplification may include attenuating or suppressing one or moreaudio signals received by the microphone from directions outside theregion. In some embodiments, step 2116 may further comprise determining,based on analysis of the plurality of images, that the recognizedindividual is speaking and trigger the selective conditioning based onthe determination that the recognized individual is speaking. Forexample, the determination that the recognized individual is speakingmay be based on detected movement of the recognized individual's lips.In some embodiments, selective conditioning may be based on furtheranalysis of the captured images as described above, for example, basedon the direction or proximity of the recognized individual, the identityof the recognized individual, the behavior of other individuals, etc.

In step 2118, process 2100 may include causing transmission of the atleast one conditioned audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user. The conditioned audiosignal, for example, may be transmitted to hearing interface device1710, which may provide sound corresponding to the audio signal to user100. The processor performing process 2100 may further be configured tocause transmission to the hearing interface device of one or more audiosignals representative of background noise, which may be attenuatedrelative to the at least one conditioned audio signal. For example,processor 210 may be configured to transmit audio signals correspondingto sounds 2020, 2021, and 2022. The signal associated with 2020,however, may be amplified in relation to sounds 2021 and 2022 based on adetermination that sound 2020 is within region 2030. In someembodiments, hearing interface device 1710 may include a speakerassociated with an earpiece. For example, hearing interface device 1710may be inserted at least partially into the ear of the user forproviding audio to the user. Hearing interface device may also beexternal to the ear, such as a behind-the-ear hearing device, one ormore headphones, a small portable speaker, or the like. In someembodiments, hearing interface device may include a bone conductionmicrophone, configured to provide an audio signal to user throughvibrations of a bone of the user's head. Such devices may be placed incontact with the exterior of the user's skin, or may be implantedsurgically and attached to the bone of the user.

In addition to recognizing voices of individuals speaking to user 100,the systems and methods described above may also be used to recognizethe voice of user 100. For example, voice recognition unit 2041 may beconfigured to analyze audio signals representative of sounds collectedfrom the user's environment to recognize the voice of user 100. Similarto the selective conditioning of the voice of recognized individuals,the voice of user 100 may be selectively conditioned. For example,sounds may be collected by microphone 1720, or by a microphone ofanother device, such as a mobile phone (or a device linked to a mobilephone). Audio signals corresponding to the voice of user 100 may beselectively transmitted to a remote device, for example, by amplifyingthe voice of user 100 and/or attenuating or eliminating altogethersounds other than the user's voice. Accordingly, a voiceprint of one ormore users of apparatus 110 may be collected and/or stored to facilitatedetection and/or isolation of the user's voice, as described in furtherdetail above.

FIG. 22 is a flowchart showing an exemplary process 2200 for selectivelytransmitting audio signals associated with a voice of a recognized userconsistent with disclosed embodiments. Process 2200 may be performed byone or more processors associated with apparatus 110, such as processor210.

In step 2210, process 2200 may include receiving audio signalsrepresentative of sounds captured by a microphone. For example,apparatus 110 may receive audio signals representative of sounds 2020,2021, and 2022, captured by microphone 1720. Accordingly, the microphonemay include a directional microphone, a microphone array, a multi-portmicrophone, or various other types of microphones, as described above.In step 2212, process 2200 may include identifying, based on analysis ofthe received audio signals, one or more voice audio signalsrepresentative of a recognized voice of the user. For example, the voiceof the user may be recognized based on a voiceprint associated with theuser, which may be stored in memory 550, database 2050, or othersuitable locations. Processor 210 may recognize the voice of the user,for example, using voice recognition component 2041. Processor 210 mayseparate an ongoing voice signal associated with the user almost in realtime, e.g. with a minimal delay, using a sliding time window. The voicemay be separated by extracting spectral features of an audio signalaccording to the methods described above.

In step 2214, process 2200 may include causing transmission, to aremotely located device, of the one or more voice audio signalsrepresentative of the recognized voice of the user. The remotely locateddevice may be any device configured to receive audio signals remotely,either by a wired or wireless form of communication. In someembodiments, the remotely located device may be another device of theuser, such as a mobile phone, an audio interface device, or another formof computing device. In some embodiments, the voice audio signals may beprocessed by the remotely located device and/or transmitted further. Instep 2216, process 2200 may include preventing transmission, to theremotely located device, of at least one background noise audio signaldifferent from the one or more voice audio signals representative of arecognized voice of the user. For example, processor 210 may attenuateand/or eliminate audio signals associated with sounds 2020, 2021, or2023, which may represent background noise. The voice of the user may beseparated from other noises using the audio processing techniquesdescribed above.

In an exemplary illustration, the voice audio signals may be captured bya headset or other device worn by the user. The voice of the user may berecognized and isolated from the background noise in the environment ofthe user. The headset may transmit the conditioned audio signal of theuser's voice to a mobile phone of the user. For example, the user may beon a telephone call and the conditioned audio signal may be transmittedby the mobile phone to a recipient of the call. The voice of the usermay also be recorded by the remotely located device. The audio signal,for example, may be stored on a remote server or other computing device.In some embodiments, the remotely located device may process thereceived audio signal, for example, to convert the recognized user'svoice into text.

Lip-Tracking Hearing Aid

Consistent with the disclosed embodiments, a hearing aid system mayselectively amplify audio signals based on tracked lip movements. Thehearing aid system analyzes captured images of the environment of a userto detect lips of an individual and track movement of the individual'slips. The tracked lip movements may serve as a cue for selectivelyamplifying audio received by the hearing aid system. For example, voicesignals determined to sync with the tracked lip movements or that areconsistent with the tracked lip movements may be selectively amplifiedor otherwise conditioned. Audio signals that are not associated with thedetected lip movement may be suppressed, attenuated, filtered or thelike.

User 100 may wear a hearing aid device consistent with the camera-basedhearing aid device discussed above. For example, the hearing aid devicemay be hearing interface device 1710, as shown in FIG. 17A. Hearinginterface device 1710 may be any device configured to provide audiblefeedback to user 100. Hearing interface device 1710 may be placed in oneor both ears of user 100, similar to traditional hearing interfacedevices. As discussed above, hearing interface device 1710 may be ofvarious styles, including in-the-canal, completely-in-canal, in-the-ear,behind-the-ear, on-the-ear, receiver-in-canal, open fit, or variousother styles. Hearing interface device 1710 may include one or morespeakers for providing audible feedback to user 100, microphones fordetecting sounds in the environment of user 100, internal electronics,processors, memories, etc. In some embodiments, in addition to orinstead of a microphone, hearing interface device 1710 may comprise oneor more communication units, and one or more receivers for receivingsignals from apparatus 110 and transferring the signals to user 100.Hearing interface device 1710 may correspond to feedback outputting unit230 or may be separate from feedback outputting unit 230 and may beconfigured to receive signals from feedback outputting unit 230.

In some embodiments, hearing interface device 1710 may comprise a boneconduction headphone 1711, as shown in FIG. 17A. Bone conductionheadphone 1711 may be surgically implanted and may provide audiblefeedback to user 100 through bone conduction of sound vibrations to theinner ear. Hearing interface device 1710 may also comprise one or moreheadphones (e.g., wireless headphones, over-ear headphones, etc.) or aportable speaker carried or worn by user 100. In some embodiments,hearing interface device 1710 may be integrated into other devices, suchas a Bluetooth™ headset of the user, glasses, a helmet (e.g., motorcyclehelmets, bicycle helmets, etc.), a hat, etc.

Hearing interface device 1710 may be configured to communicate with acamera device, such as apparatus 110. Such communication may be througha wired connection, or may be made wirelessly (e.g., using a Bluetooth™,NFC, or forms of wireless communication). As discussed above, apparatus110 may be worn by user 100 in various configurations, including beingphysically connected to a shirt, necklace, a belt, glasses, a wriststrap, a button, or other articles associated with user 100. In someembodiments, one or more additional devices may also be included, suchas computing device 120. Accordingly, one or more of the processes orfunctions described herein with respect to apparatus 110 or processor210 may be performed by computing device 120 and/or processor 540.

As discussed above, apparatus 110 may comprise at least one microphoneand at least one image capture device. Apparatus 110 may comprisemicrophone 1720, as described with respect to FIG. 17B. Microphone 1720may be configured to determine a directionality of sounds in theenvironment of user 100. For example, microphone 1720 may comprise oneor more directional microphones, a microphone array, a multi-portmicrophone, or the like. Processor 210 may be configured to distinguishsounds within the environment of user 100 and determine an approximatedirectionality of each sound. For example, using an array of microphones1720, processor 210 may compare the relative timing or amplitude of anindividual sound among the microphones 1720 to determine adirectionality relative to apparatus 100. Apparatus 110 may comprise oneor more cameras, such as camera 1730, which may correspond to imagesensor 220. Camera 1730 may be configured to capture images of thesurrounding environment of user 100. Apparatus 110 may also use one ormore microphones of hearing interface device 1710 and, accordingly,references to microphone 1720 used herein may also refer to a microphoneon hearing interface device 1710.

Processor 210 (and/or processors 210 a and 210 b) may be configured todetect a mouth and/or lips associated with an individual within theenvironment of user 100. FIGS. 23A and 23B show an exemplary individual2310 that may be captured by camera 1730 in the environment of a userconsistent with the present disclosure. As shown in FIG. 23, individual2310 may be physically present with the environment of user 100.Processor 210 may be configured to analyze images captured by camera1730 to detect a representation of individual 2310 in the imagesProcessor 210 may use a facial recognition component, such as facialrecognition component 2040, described above, to detect and identifyindividuals in the environment of user 100. Processor 210 may beconfigured to detect one or more facial features of user 2310, includinga mouth 2311 of individual 2310. Accordingly, processor 210 may use oneor more facial recognition and/or feature recognition techniques, asdescribed further below.

In some embodiments, processor 210 may detect a visual representation ofindividual 2310 from the environment of user 100, such as a video ofuser 2310. As shown in FIG. 23B, user 2310 may be detected on thedisplay of a display device 2301. Display device 2301 may be any devicecapable of displaying a visual representation of an individual. Forexample, display device may be a personal computer, a laptop, a mobilephone, a tablet, a television, a movie screen, a handheld gaming device,a video conferencing device (e.g., Facebook Portal™, etc.), a babymonitor, etc. The visual representation of individual 2310 may be a livevideo feed of individual 2310, such as a video call, a conference call,a surveillance video, etc. In other embodiments, the visualrepresentation of individual 2310 may be a prerecorded video or image,such as a video message, a television program, or a movie. Processor 210may detect one or more facial features based on the visualrepresentation of individual 2310, including a mouth 2311 of individual2310.

FIG. 23C illustrates an exemplary lip-tracking system consistent withthe disclosed embodiments. Processor 210 may be configured to detect oneor more facial features of individual 2310, which may include, but isnot limited to the individual's mouth 2311. Accordingly, processor 210may use one or more image processing techniques to recognize facialfeatures of the user, such as convolutional neural networks (CNN),scale-invariant feature transform (SIFT), histogram of orientedgradients (HOG) features, or other techniques. In some embodiments,processor 210 may be configured to detect one or more points 2320associated with the mouth 2311 of individual 2310. Points 2320 mayrepresent one or more characteristic points of an individual's mouth,such as one or more points along the individual's lips or the corner ofthe individual's mouth. The points shown in FIG. 23C are forillustrative purposes only and it is understood that any points fortracking the individual's lips may be determined or identified via oneor more image processing techniques. Points 2320 may be detected atvarious other locations, including points associated with theindividual's teeth, tongue, cheek, chin, eyes, etc. Processor 210 maydetermine one or more contours of mouth 2311 (e.g., represented by linesor polygons) based on points 2320 or based on the captured image. Thecontour may represent the entire mouth 2311 or may comprise multiplecontours, for example including a contour representing an upper lip anda contour representing a lower lip. Each lip may also be represented bymultiple contours, such as a contour for the upper edge and a contourfor the lower edge of each lip. Processor 210 may further use variousother techniques or characteristics, such as color, edge, shape ormotion detection algorithms to identify the lips of individual 2310. Theidentified lips may be tracked over multiple frames or images. Processor210 may use one or more video tracking algorithms, such as mean-shifttracking, contour tracking (e.g., a condensation algorithm), or variousother techniques. Accordingly, processor 210 may be configured to trackmovement of the lips of individual 2310 in real time.

The tracked lip movement of individual 2310 may be used to separate ifrequired, and selectively condition one or more sounds in theenvironment of user 100. FIG. 24 is a schematic illustration showing anexemplary environment 2400 for use of a lip-tracking hearing aidconsistent with the present disclosure. Apparatus 110, worn by user 100may be configured to identify one or more individuals within environment2400. For example, apparatus 110 may be configured to capture one ormore images of the surrounding environment 2400 using camera 1730. Thecaptured images may include a representation of individuals 2310 and2410, who may be present in environment 2400. Processor 210 may beconfigured to detect a mouth of individuals 2310 and 2410 and tracktheir respective lip movements using the methods described above. Insome embodiments, processor 210 may further be configured to identifyindividuals 2310 and 2410, for example, by detecting facial features ofindividuals 2310 and 2410 and comparing them to a database, as discussedpreviously.

In addition to detecting images, apparatus 110 may be configured todetect one or more sounds in the environment of user 100. For example,microphone 1720 may detect one or more sounds 2421, 2422, and 2423within environment 2400. In some embodiments, the sounds may representvoices of various individuals. For example, as shown in FIG. 24, sound2421 may represent a voice of individual 2310 and sound 2422 mayrepresent a voice of individual 2410. Sound 2423 may representadditional voices and/or background noise within environment 2400.Processor 210 may be configured to analyze sounds 2421, 2422, and 2423to separate and identify audio signals associated with voices. Forexample, processor 210 may use one or more speech or voice activitydetection (VAD) algorithms and/or the voice separation techniquesdescribed above. When there are multiple voices detected in theenvironment, processor 210 may isolate audio signals associated witheach voice. In some embodiments, processor 210 may perform furtheranalysis on the audio signal associated the detected voice activity torecognize the speech of the individual. For example, processor 210 mayuse one or more voice recognition algorithms (e.g., Hidden MarkovModels, Dynamic Time Warping, neural networks, or other techniques) torecognize the voice of the individual. Processor 210 may also beconfigured to recognize the words spoken by individual 2310 usingvarious speech-to-text algorithms. In some embodiments, instead of usingmicrophone 1710, apparatus 110 may receive audio signals from anotherdevice through a communication component, such as wireless transceiver530. For example, if user 100 is on a video call, apparatus 110 mayreceive an audio signal representing a voice of user 2310 from displaydevice 2301 or another auxiliary device.

Processor 210 may determine, based on lip movements and the detectedsounds, which individuals in environment 2400 are speaking. For example,processor 2310 may track lip movements associated with mouth 2311 todetermine that individual 2310 is speaking. A comparative analysis maybe performed between the detected lip movement and the received audiosignals. In some embodiments, processor 210 may determine thatindividual 2310 is speaking based on a determination that mouth 2311 ismoving at the same time as sound 2421 is detected. For example, when thelips of individual 2310 stop moving, this may correspond with a periodof silence or reduced volume in the audio signal associated with sound2421. In some embodiments, processor 210 may be configured to determinewhether specific movements of mouth 2311 correspond to the receivedaudio signal. For example, processor 210 may analyze the received audiosignal to identify specific phonemes, phoneme combinations or words inthe received audio signal. Processor 210 may recognize whether specificlip movements of mouth 2311 correspond to the identified words orphonemes. Various machine learning or deep learning techniques may beimplemented to correlate the expected lip movements to the detectedaudio. For example, a training data set of known sounds andcorresponding lip movements may be fed to a machine learning algorithmto develop a model for correlating detected sounds with expected lipmovements. Other data associated with apparatus 110 may further be usedin conjunction with the detected lip movement to determine and/or verifywhether individual 2310 is speaking, such as a look direction of user100 or individual 2310, a detected identity of user 2310, a recognizedvoiceprint of user 2310, etc.

Based on the detected lip movement, processor 210 may cause selectiveconditioning of audio associated with individual 2310. The conditioningmay include amplifying audio signals determined to correspond to sound2421 (which may correspond to a voice of individual 2310) relative toother audio signals. In some embodiments, amplification may beaccomplished digitally, for example by processing audio signalsassociated with sound 2421 relative to other signals. Additionally, oralternatively, amplification may be accomplished by changing one or moreparameters of microphone 1720 to focus on audio sounds associated withindividual 2310. For example, microphone 1720 may be a directionalmicrophone and processor 210 may perform an operation to focusmicrophone 1720 on sound 2421. Various other techniques for amplifyingsound 2421 may be used, such as using a beamforming microphone array,acoustic telescope techniques, etc. The conditioned audio signal may betransmitted to hearing interface device 1710, and thus may provide user100 with audio conditioned based on the individual who is speaking.

In some embodiments, selective conditioning may include attenuation orsuppressing one or more audio signals not associated with individual2310, such as sounds 2422 and 2423. Similar to amplification of sound2421, attenuation of sounds may occur through processing audio signals,or by varying one or more parameters associated with microphone 1720 todirect focus away from sounds not associated with individual 2310.

In some embodiments, conditioning may further include changing a tone ofone or more audio signals corresponding to sound 2421 to make the soundmore perceptible to user 100. For example, user 100 may have lessersensitivity to tones in a certain range and conditioning of the audiosignals may adjust the pitch of sound 2421. For example, user 100 mayexperience hearing loss in frequencies above 10 kHz and processor 210may remap higher frequencies (e.g., at 15 kHz) to 10 kHz. In someembodiments processor 210 may be configured to change a rate of speechassociated with one or more audio signals. Processor 210 may beconfigured to vary the rate of speech of individual 2310 to make thedetected speech more perceptible to user 100. If speech recognition hasbeen performed on the audio signal associated with sound 2421,conditioning may further include modifying the audio signal based on thedetected speech. For example, processor 210 may introduce pauses orincrease the duration of pauses between words and/or sentences, whichmay make the speech easier to understand. Various other processing maybe performed, such as modifying the tone of sound 2421 to maintain thesame pitch as the original audio signal, or to reduce noise within theaudio signal.

The conditioned audio signal may then be transmitted to hearinginterface device 1710 and then produced for user 100. Thus, in theconditioned audio signal, sound 2421 (may be louder and/or more easilydistinguishable than sounds 2422 and 2423.

Processor 210 may be configured to selectively condition multiple audiosignals based on which individuals associated with the audio signals arecurrently speaking. For example, individual 2310 and individual 2410 maybe engaged in a conversation within environment 2400 and processor 210may be configured to transition from conditioning of audio signalsassociated with sound 2421 to conditioning of audio signals associatedwith sound 2422 based on the respective lip movements of individuals2310 and 2410. For example, lip movements of individual 2310 mayindicate that individual 2310 has stopped speaking or lip movementsassociated with individual 2410 may indicate that individual 2410 hasstarted speaking. Accordingly, processor 210 may transition betweenselectively conditioning audio signals associated with sound 2421 toaudio signals associated with sound 2422. In some embodiments, processor210 may be configured to process and/or condition both audio signalsconcurrently but only selectively transmit the conditioned audio tohearing interface device 1710 based on which individual is speaking.Where speech recognition is implemented, processor 210 may determineand/or anticipate a transition between speakers based on the context ofthe speech. For example, processor 210 may analyze audio signalsassociate with sound 2421 to determine that individual 2310 has reachedthe end of a sentence or has asked a question, which may indicateindividual 2310 has finished or is about to finish speaking.

In some embodiments, processor 210 may be configured to select betweenmultiple active speakers to selectively condition audio signals. Forexample, individuals 2310 and 2410 may both be speaking at the same timeor their speech may overlap during a conversation. Processor 210 mayselectively condition audio associated with one speaking individualrelative to others. This may include giving priority to a speaker whohas started but not finished a word or sentence or has not finishedspeaking altogether when the other speaker started speaking. Thisdetermination may also be driven by the context of the speech, asdescribed above.

Various other factors may also be considered in selecting among activespeakers. For example, a look direction of the user may be determinedand the individual in the look direction of the user may be given higherpriority among the active speakers. Priority may also be assigned basedon the look direction of the speakers. For example, if individual 2310is looking at user 100 and individual 2410 is looking elsewhere, audiosignals associated with individual 2310 may be selectively conditioned.In some embodiments, priority may be assigned based on the relativebehavior of other individuals in environment 2400. For example, if bothindividual 2310 and individual 2410 are speaking and more otherindividuals are looking at individual 2410 than individual 2310, audiosignals associated with individual 2410 may be selectively conditionedover those associated with individual 2310. In embodiments where theidentity of the individuals is determined, priority may be assignedbased on the relative status of the speakers, as discussed previously ingreater detail. User 100 may also provide input into which speakers areprioritized through predefined settings or by actively selecting whichspeaker to focus on.

Processor 210 may also assign priority based on how the representationof individual 2310 is detected. While individuals 2310 and 2410 areshown to be physically present in environment 2400, one or moreindividuals may be detected as visual representations of the individual(e.g., on a display device) as shown in FIG. 23B. Processor 210 mayprioritize speakers based on whether or not they are physically presentin environment 2400. For example, processor 210 may prioritize speakerswho are physically present over speakers on a display. Alternatively,processor 210 may prioritize a video over speakers in a room, forexample, if user 100 is on a video conference or if user 100 is watchinga movie. The prioritized speaker or speaker type (e.g. present or not)may also be indicated by user 100, using a user interface associatedwith apparatus 110.

FIG. 25 is a flowchart showing an exemplary process 2500 for selectivelyamplifying audio signals based on tracked lip movements consistent withdisclosed embodiments. Process 2500 may be performed by one or moreprocessors associated with apparatus 110, such as processor 210. Theprocessor(s) may be included in the same common housing as microphone1720 and camera 1730, which may also be used for process 2500. In someembodiments, some or all of process 2500 may be performed on processorsexternal to apparatus 110, which may be included in a second housing.For example, one or more portions of process 2500 may be performed byprocessors in hearing interface device 1710, or in an auxiliary device,such as computing device 120 or display device 2301. In suchembodiments, the processor may be configured to receive the capturedimages via a wireless link between a transmitter in the common housingand receiver in the second housing.

In step 2510, process 2500 may include receiving a plurality of imagescaptured by a wearable camera from an environment of the user. Theimages may be captured by a wearable camera such as camera 1730 ofapparatus 110. In step 2520, process 2500 may include identifying arepresentation of at least one individual in at least one of theplurality of images. The individual may be identified using variousimage detection algorithms, such as Haar cascade, histograms of orientedgradients (HOG), deep convolution neural networks (CNN), scale-invariantfeature transform (SIFT), or the like. In some embodiments, processor210 may be configured to detect visual representations of individuals,for example from a display device, as shown in FIG. 23B.

In step 2530, process 2500 may include identifying at least one lipmovement or lip position associated with a mouth of the individual,based on analysis of the plurality of images. Processor 210 may beconfigured to identify one or more points associated with the mouth ofthe individual. In some embodiments, processor 210 may develop a contourassociated with the mouth of the individual, which may define a boundaryassociated with the mouth or lips of the individual. The lips identifiedin the image may be tracked over multiple frames or images to identifythe lip movement. Accordingly, processor 210 may use various videotracking algorithms, as described above.

In step 2540, process 2500 may include receiving audio signalsrepresentative of the sounds captured by a microphone from theenvironment of the user. For example, apparatus 110 may receive audiosignals representative of sounds 2421, 2422, and 2423 captured bymicrophone 1720. In step 2550, process 2500 may include identifying,based on analysis of the sounds captured by the microphone, a firstaudio signal associated with a first voice and a second audio signalassociated with a second voice different from the first voice. Forexample, processor 210 may identify an audio signal associated withsounds 2421 and 2422, representing the voice of individuals 2310 and2410, respectively. Processor 210 may analyze the sounds received frommicrophone 1720 to separate the first and second voices using anycurrently known or future developed techniques or algorithms. Step 2550may also include identifying additional sounds, such as sound 2423 whichmay include additional voices or background noise in the environment ofthe user. In some embodiments, processor 210 may perform furtheranalysis on the first and second audio signals, for example, bydetermining the identity of individuals 2310 and 2410 using availablevoiceprints thereof. Alternatively, or additionally, processor 210 mayuse speech recognition tools or algorithms to recognize the speech ofthe individuals.

In step 2560, process 2500 may include causing selective conditioning ofthe first audio signal based on a determination that the first audiosignal is associated with the identified lip movement associated withthe mouth of the individual. Processor 210 may compare the identifiedlip movement with the first and second audio signals identified in step2550. For example, processor 210 may compare the timing of the detectedlip movements with the timing of the voice patterns in the audiosignals. In embodiments where speech is detected, processor 210 mayfurther compare specific lip movements to phonemes or other featuresdetected in the audio signal, as described above. Accordingly, processor210 may determine that the first audio signal is associated with thedetected lip movements and is thus associated with an individual who isspeaking.

Various forms of selective conditioning may be performed, as discussedabove. In some embodiments, conditioning may include changing the toneor playback speed of an audio signal. For example, conditioning mayinclude remapping the audio frequencies or changing a rate of speechassociated with the audio signal. In some embodiments, the conditioningmay include amplification of a first audio signal relative to otheraudio signals. Amplification may be performed by various means, such asoperation of a directional microphone, varying one or more parametersassociated with the microphone, or digitally processing the audiosignals. The conditioning may include attenuating or suppressing one ormore audio signals that are not associated with the detected lipmovement. The attenuated audio signals may include audio signalsassociated with other sounds detected in the environment of the user,including other voices such as a second audio signal. For example,processor 210 may selectively attenuate the second audio signal based ona determination that the second audio signal is not associated with theidentified lip movement associated with the mouth of the individual. Insome embodiments, the processor may be configured to transition fromconditioning of audio signals associated with a first individual toconditioning of audio signals associated with a second individual whenidentified lip movements of the first individual indicates that thefirst individual has finished a sentence or has finished speaking.

In step 2570, process 2500 may include causing transmission of theselectively conditioned first audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user. The conditioned audiosignal, for example, may be transmitted to hearing interface device1710, which may provide sound corresponding to the first audio signal touser 100. Additional sounds such as the second audio signal may also betransmitted. For example, processor 210 may be configured to transmitaudio signals corresponding to sounds 2421, 2422, and 2423. The firstaudio signal, which may be associated with the detected lip movement ofindividual 2310, may be amplified, however, in relation to sounds 2422and 2423 as described above. In some embodiments, hearing interface 1710device may include a speaker associated with an earpiece. For example,hearing interface device may be inserted at least partially into the earof the user for providing audio to the user. Hearing interface devicemay also be external to the ear, such as a behind-the-ear hearingdevice, one or more headphones, a small portable speaker, or the like.In some embodiments, hearing interface device may include a boneconduction microphone, configured to provide an audio signal to userthrough vibrations of a bone of the user's head. Such devices may beplaced in contact with the exterior of the user's skin, or may beimplanted surgically and attached to the bone of the user.

Selective Amplification of Speaker of Interest

The disclosed systems and methods may enable a hearing aid system toselectively amplify an audio signal and transmit the amplified audiosignal to a hearing interface device configured to provide sound to anear of a user. For example, the system may recognize multiple speakersthrough analysis of captured images, but may selectively amplify one ofthe voices of the detected speakers. The voice selected foramplification may be based on a hierarchy or other suitabledifferentiator. In one example, the voice of the speaker whom the useris looking at may be amplified. In another example, the voice of aspeaker detected to be looking toward the user may be amplified. Inanother example, in case of speech overlap, the voice of a speaker whoalready started speaking but has not finished when another speaker hasstarted speaking may be selected.

FIG. 26 illustrates a user wearing an exemplary hearing aid system. User2601 may wear a wearable device 2631. Wearable device 2631 may includean image sensor configured to capture images of the environment of user2601. As illustrated in FIG. 26, a first individual 2611 may stand infront of user 2601 and look in the direction of user 2601. In addition,a second individual 2612 may also stand in front of user 2601, but lookin a direction away from user 2601. The image sensor of wearable device2631 may capture one or more images including first individual 2611 andsecond individual 2612.

FIG. 27 illustrates an exemplary image 2700 of the environment of user2601 illustrated in FIG. 26 as may be captured by an image sensor ofwearable device 2631. Image 2700 may include a presentation 2711 offirst individual 2611 and a presentation 2712 of second individual 2612.

Wearable device 2631 may also include at least one processor configuredto analyze the images captured by the image sensor. The processor mayalso identify one or more individuals included in the images, based onthe image analysis. For example, the processor may receive image 2700(illustrated in FIG. 27) from the image sensor. The processor may alsoidentify first individual 2611 and second individual 2612 included inthe image.

Wearable device 2631 may further include at least one microphoneconfigured to receive one or more audio signals from the environment ofuser 2601. For example, the microphone may be configured to receive (ordetect) a first audio signal associated with the voice of the firstindividual 2611 and a second audio signal associated with the voice ofthe second individual 2612.

The processor may detect at least one amplification criteria indicativeof a voice amplification priority between the first individual and thesecond individual. Some amplification criteria may be static, whileothers may be dynamic. For example, based on the analysis of image 2700,the processor may detect that first individual 2611 is looking in thedirection of user 2601 and second individual 2612 is looking in adirection away user 2601, which may indicate that the voiceamplification priority of first individual 2611 should be higher thansecond individual 2612. The processor may also selectively amplify thefirst audio signal, based on the voice amplification priority.

Wearable device 2631 may further include a hearing interface device suchas hearing interface device 1710, configured to receive audio signalsand provide sound to an ear of user 2601. For example, the hearinginterface device may receive the amplified first audio signal andprovide sound to user 2601 based on the amplified first audio signal. Insome embodiments, the hearing interface device may receive the amplifiedfirst audio signal and unprocessed second audio signal, and providesound to user 2601 based on the amplified first audio signal and secondaudio signal.

FIG. 28 is a flowchart of an exemplary process 2800 for selectivelyamplifying an audio signal. At step 2801, the hearing aid system (e.g.,apparatus 110) may receive a plurality of images of an environment ofthe user. For example, the hearing aid system may include a processor(e.g., processor 210) configured to receive images of the environment ofthe user captured by an image sensor (e.g., image sensor 220). In someembodiments, the image sensor may be part of a camera included thehearing aid system. By way of example, as illustrated in FIG. 26, user2601 may wear a wearable device 2631 that may include an image sensorconfigured to capture images of the environment of the user. Theprocessor of the hearing aid system may receive the images from wearabledevice 2631.

In some embodiments, the processor may be configured to control theimage sensor to capture images. For example, the processor may detect agesture performed by the user (a finger-pointing gesture) and controlthe image sensor to capture images based on the detected gesture (e.g.,adjusting the field of view of the image sensor based on the directionof the finger-pointing gesture). As another example, the hearing aidsystem may include a microphone configured to detect (or receive) audiosignals from the environment of the user. The processor may receive theaudio signals from the microphone and detect a voice by one or moreindividuals nearby. The processor may control the image sensor tocapture images if a voice is detected.

In some embodiments, the processor may receive data from and transmitdata to the image sensor over one or more networks via any knownwireless standard (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or vianear-filed capacitive coupling, other short-range wireless techniques,or via a wired connection. For example, the processor may also beconfigured to receive data (e.g., the captured images, etc.) from theimage sensor via a wireless link between a transmitter in a housing inwhich the image sensor is included and a receiver in a housing in whichthe processor is included.

At step 2803, the processor may analyze one or more images received fromthe image sensor and identify that one or more individuals are includedin the images. For example, as illustrated in FIG. 26, twoindividuals—first individual 2611 and second individual 2612—stand infront of user 2601 (first individual 2611 may stand closer to user 2601than second individual 2612 does). The image sensor may be configured tocapture image 2700 (illustrated in FIG. 27) of the environment of user2601, including first individual 2611 and second individual 2612. Theprocessor may analyze image 2700 and identify in image 2700 arepresentation 2711 of first individual 2611 and representation 2712 ofsecond individual 2612. Representation 2711 of first individual 2611 mayappear bigger than representation 2712 of second individual 2612 sincefirst individual 2611 may stand closer to user 2601 than secondindividual 2612 does. In some embodiments, the processor may identifyone or more individuals based on object recognition techniques (e.g., adeep-learning algorithm for recognizing objects).

In some embodiments, the processor may recognize one or more individualsincluded in the images. For example, the processor may recognize one ofthe individuals is a family member or a friend, based on a humanrecognition technique (e.g., a deep-learning algorithm for recognizingan individual). In some embodiments, the processor may be configured toretrieve information relating to the recognized individual (e.g., thename of the individual and the last time the user met the individual).The processor may also transmit the information to the user via thehearing aid interface and/or a feedback-outputting unit.

In some embodiments, the processor may be configured to determine thevisual line (or the look direction) of each of one or more identifiedindividuals and/or the user, based on the analysis of the images. Forexample, an individual may look at the user, and the processor maydetermine that the look direction of the individual is towards the user,based on the image analysis. As another example, the processor maydetermine the look direction of the user based on the image analysis.

At step 2805, the processor may receive from at least one microphone afirst audio signal associated with a voice of the first individual. Forexample, the hearing aid system may include one or more microphonesconfigured to detect (or receive) audio signals from the environment ofthe user. By way of example, wearable device 2631 (e.g., illustrated inFIG. 26) may include a microphone configured to receive a first audiosignal associated with first individual 2611 and may receive a secondaudio signal associated with the second individual 2612, who stands infront of user 2601. The processor may receive the first audio signalfrom the microphone. In some embodiments, the processor may receive datafrom the microphone over one or more networks via any known wirelessstandard (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or via near-filedcapacitive coupling, other short-range wireless techniques, or via awired connection. For example, the processor may also be configured toreceive data (e.g., the audio signals, etc.) from the microphone via awireless link between a transmitter in a housing in which the microphoneis included and a receiver in a housing in which the processor isincluded.

In some embodiments, the processor may be configured to control themicrophone to detect (or receive) audio signals and/or transmit theaudio signals to the processing device (and/or the hearing aidinterface). For example, the processor may identify one or moreindividuals based on the analysis of the audio signals. The processormay activate the microphone to receive audio signals if one or moreindividuals are identified. In some embodiments, if a speaker isrecognized and the audio signal is transmitted to the processing device,the audio signal associated with the speaker may be transmitted to theprocessing device as long as the speaker keeps speaking (or a pause isless than a threshold). In some embodiments, the audio signal associatedwith the speaker may be transmitted to the processing device as long asthe speaker keeps speaking (or a pause is less than a threshold) even ifother voices, whether recognized or not are captured, to let the usercontinuously listen to the speaker. For example, the processor may beconfigured to continue causing transmission of the first audio signal,rather than the second audio signal, to the hearing interface deviceconfigured to provide sound to an ear of the user, until a pause longerthan a predetermined length is detected in speech associated the voiceof the first individual. In some embodiments, short breaks in speaking,for example, breathing breaks or pauses for searching for a word maystill be considered as continuous speech. In some embodiments, pauses upto a predetermined length may be considered as part of continuousspeech, while longer periods may be considered as the end of a speech bythe speaker, such that other speakers can be detected or amplified oramplified to a different degree.

In some embodiments, the microphone may include a directional microphone(e.g., a bi-directional microphone, an omnidirectional microphone,etc.), a microphone array, or the like, or a combination thereof. Insome embodiments, the processor may be configured to determine thespeaking direction of each of one or more identified individuals and/orthe user, based on the audio signals received. For example, themicrophone may include one or more directional microphones, and theprocessor may be configured to determine the speak direction of anindividual based on the audio signal associated with the individual.

At step 2807, the processor may receive from the microphone a secondaudio signal associated with a voice of the second individual. By way ofexample, wearable device 2631, illustrated in FIG. 26, may include amicrophone configured to receive a second audio signal associated withsecond individual 2612, who stand in front of user 2601.

In some embodiments, the processor may be configured to receive audiosignals from the microphone and recognize the individual associated withthe audio signal received. For example, the processor may recognize theindividual based on the characteristics of the voice of the individual(e.g., the individual's voice speed, pitch, etc.). In some embodiments,the processor may be configured to retrieve information relating to therecognized individual (e.g., the name of the individual and the lasttime met the individual). The processor may also transmit theinformation to the user via the hearing aid interface and/or afeedback-outputting unit.

In some embodiments, the processor may be configured to recognize theindividual associated with the audio signal based on analysis of theaudio signal and one or more images received from the image sensor. Forexample, the processor may determine a first confidence score for theassociation of an individual and the audio signal based on the analysisof the audio signal. The processor may also determine a secondconfidence score for the association of the individual and the audiosignal based on the analysis of the one or more images received from theimage sensor (similar to the recognition process in step 2803). Theprocessor may further determine an overall confidence score based on thefirst and second confidence scores and identify the individual based onthe overall confidence score (e.g., identifying the individual if theoverall confidence score exceeds a threshold). By way of example, theprocessor may determine a first confidence score of 9 (out of 10) forthe association of a specific individual and an audio signal based onthe analysis of the audio signal. The processor may also determine asecond confidence score of 2 (out of 10) for the association of thespecific individual and the audio signal based on the analysis of one ormore images received from the image sensor. The processor may furtherdetermine an overall confidence score of 11 (i.e., 2 plus 9 out of 20 intotal) and determine that this individual is not associated with theaudio signal if the threshold is 16. As another example, the processormay determine the first confidence score of 9 and determine the secondconfidence score of 8. The processor may also determine that the overallconfidence score is 17 and recognize the individual being associatedwith the audio signal. In some embodiments, the processor may determinethe overall confidence score based a weighted first confidence scoreand/or a weighted second confidence score.

In some embodiments, the microphone and the image sensor (or thewearable camera that includes the image sensor) may be included in acommon housing. For example, wearable device 2631 illustrated in FIG. 26may include both the microphone and the wearable camera in a commonhousing. Alternatively, the microphone may be included in a housingdifferent from a housing in which the wearable camera is installed.

In some embodiments, the processor may be included in a common housingwith at least one of the microphone and the wearable camera. Forexample, the processor may also be included in a common housing in whichboth the microphone and the wearable camera are included. Alternatively,the processor may be included in a separate housing from a commonhousing where the microphone and the wearable camera are installed. Theprocessor may also be configured to receive data (e.g., the capturedimages, the detected audio signals, etc.) from the wearable cameraand/or the microphone via a wireless link between a transmitter in thecommon housing (in which the microphone and the wearable camera areincluded) and receiver in the second housing (in which the processor isincluded).

At step 2809, the processor may detect at least one amplificationcriteria indicative of a voice amplification priority between the firstindividual and the second individual. The detection of an amplificationcriteria may be based on the analysis of the received images and/oraudio signals. For example, the processor may detect that the firstindividual stands closer to the user than the second individual standsbased on, for example, the image analysis. The detection that the useris closer to the first individual than the second individual may be anamplification criteria indicative of a voice amplification priority ofthe first individual over the second individual.

In some embodiments, the amplification criteria may include the positionand/or the orientation of the user in relative to the first and/orsecond individuals, the look direction of the user, the look directionof the speaker (e.g., the first individual, the second individual,etc.), or the like, or a combination thereof. For example, if the useris detected to be facing more towards the second individual than thefirst individual (based on the analysis of the images and/or audiosignals), the processor may detect that the second individual has ahigher voice amplification priority than the first individual does.Alternatively or additionally, the amplification criteria may relate tothe identity of the first individual and/or the second individual. Forexample, the processor may identify the first individual to be a familymember (but does not recognize the second individual) and determine thatthe first individual has a higher voice amplification priority than thesecond individual.

In some embodiments, an amplification criteria may include the lookdirection of the user, and the voice amplification priority between thefirst and second individuals may be determined based on whether the lookdirection of the user correlates with the first individual or with thesecond individual. For example, the processor may determine the lookdirection of the user based on the analysis of the images captured bythe image sensor, and determine whether the user looks towards more tothe first individual or the second individual. As another example, theprocessor may be configured to detect the look direction of the user bydetecting a representation of the user's chin in at least one of theimages and determining the look based on a detected direction associatedwith the user's chin. If the look direction of the user correlates withthe first individual more than the second individual, the processor maydetermine that the first individual has a higher voice amplificationpriority than the second individual.

In some embodiments, an amplification criteria may include the lookdirection of the speaker (e.g., the first individual, the secondindividual, etc.), and voice amplification priority between the firstand second individuals may be determined based on whether the firstindividual or the second individual is looking in the direction of theuser based on the look direction of the speaker. For example, if theprocessor determines that the first individual is looking in thedirection of the user based on the speaker look of the first individualand the second individual is looking in a direction away from the user,the first individual may have a higher voice amplification priority thanthe second individual. On the other hand, if the processor determinesthat the second individual is looking in the direction of the user andthe first individual is looking in a direction away from the user, thesecond individual may have a higher voice amplification priority thanthe first individual. In some embodiments, the processor may beconfigured to detect the look direction of the speaker based on his orher facial characteristics (e.g., the eyes, the orientation of the face,etc.) determined according to the images captured by the image sensor.

In some embodiments, amplification criteria may include a speakingcontinuity indicating that a speaker who already started speaking buthas not finished when another speaker has started speaking. For example,the first individual already started speaking but has not finished whenthe second individual has started speaking. The processor may determinethat the first individual may have a speaking continuity and have ahigher amplification priority than the second individual.

In some embodiments, amplification criteria may include a relationshipbetween the user and one of the first individual and the secondindividual. For example, the processor may identify the first individualand/or the second individual (as described elsewhere in this disclosure)and determine a relationship between the user and one of the firstindividual and the second individual. By way of example, the processormay determine that one of the individuals is a family member or a friendof the user, and determine the amplification priority based on thedetermined relationship. Exemplary relationship may include familymember, friend, acquaintance, colleague, stranger, or the like, or acombination thereof. Alternatively or additionally, the amplificationcriteria may include a relationship between the first and secondindividuals. For example, the processor may determine that the firstindividual is a supervisor of the second individual (i.e., a type ofrelationship). Alternatively or additionally, the amplification criteriamay include a relationship among the user, the first individual, and thesecond individual. For example, the processor may determine that thefirst individual is a supervisor of the user and the second individual.In some embodiments, the processor may determine the amplificationpriority based on the closeness of the relationship of the firstindividual and/or the second individual with the user. For example, theprocessor may determine that the first individual is an immediate familymember of the user and the second individual is a friend of the user,and determine that the first individual is closer (in terms ofrelationship) to the user than the second individual to the user. Theprocessor may also determine that the first individual has a higheramplification priority than the second individual. Alternatively oradditionally, the processor may determine the amplification prioritybased on the hierarchy of the determined relationship. For example, theprocessor may determine that the first individual is a supervisor of thesecond individual (i.e., a type of relationship) and determined that thefirst individual has a higher amplification priority than the secondindividual.

At step 2811, the processor may selectively amplify the first audiosignal or the second audio signal, based on the voice amplificationpriority. For example, if the first audio signal has a higher voiceamplification priority than the second audio signal, the processor mayamplify the first audio signal. Similarly, if the second audio signalhas a higher voice amplification priority than the first audio signal,the processor may amplify the second audio signal.

In some embodiments, the processor may amplify an audio signal (thefirst audio signal or the second audio signal) to a predetermined soundlevel. Alternatively or additionally, the processor may amplify an audiosignal by increasing the sound level by a percentage. Alternatively oradditionally, while amplifying an audio signal, the processor may beconfigured to attenuating one or more other audio signals (by, forexample, decreasing the sound level of the other signal(s) to apredetermined sound level or by a predetermined percentage). Forexample, if the first audio signal has a higher voice amplificationpriority than the second audio signal, the processor may be configuredto amplify the first audio signal by 50% and attenuate the second audiosignal by 50%.

In some embodiments, the hearing aid system may include an audioamplification circuit configured to selectively amplify an audio signal.The audio amplification circuit may receive inputs from two or moreinput audio transducers. For example, a first input audio transducer mayreceive the first audio signal, and a second input audio transducer mayreceive the second audio signal. The processor may cause the audioamplification circuit to amplify one of the first audio signal or thesecond audio signal, based on their voice amplification priorities.Alternatively or additionally, the processor may cause the audioamplification circuit to attenuate the audio signal that has a lowervoice amplification priority.

At step 2813, the processor may cause transmission of the selectivelyamplified audio signal to a hearing interface device. For example, theprocessor may cause a transmitter to transmit the amplified audio signalto a hearing interface device via a wireless network (e.g., cellular,Wi-Fi, Bluetooth®, etc.), or via near-filed capacitive coupling, othershort-range wireless techniques, or via a wired connection.Alternatively or additionally, the processor may cause transmission ofthe unprocessed audio signal(s) (and/or the selectively attenuated audiosignal) to the hearing interface device.

The hearing interface device may also be configured to deliver sound toan ear of the user based on the amplified audio signal. For example, thehearing interface device may receive the amplified audio signal (e.g.,amplified first audio signal) and deliver sound to an ear of the userbased on the amplified audio signal. In some embodiments, the hearinginterface device may also receive one or more unprocessed audio signalsand/or one or more attenuated audio signal. For example, the hearinginterface device may receive amplified first audio signal andunprocessed second audio signal. The hearing interface device maydeliver sound based on the amplified first audio signal and second audiosignal.

In some embodiments, a hearing interface device may include a speakerassociated with an earpiece. For example, the hearing interface devicemay include an in-ear earphone. As another example, the hearinginterface device may include a speaker included in a wearable device(e.g., wearable device 2631). In some embodiments, the hearing interfacedevice may include an earphone, a headphone, a speaker, or the like, ora combination thereof.

In some embodiments, the hearing interface device may include a boneconduction microphone.

Differential Amplification Relative to Voice of Speakerphone User

The disclosed systems and methods may enable a hearing aid system todetermine through image analysis that at least one speaker in a group isparticipating in a group meeting via speakerphone (e.g., by receiving atleast one voice signal even where image analysis indicates no visiblespeakers are present in a group). Such a voice signal may originate froma person participating in a meeting by speakerphone or from a personoutside of the field-of-view (FOV) of a wearable camera, for example aspeaker sitting in the back seat of a car while the wearable camera isfront-facing. In such scenarios, the voice signal may be weaker thanvoice signals received from individuals physically present in a group orin front of the user (e.g., with an unimpeded path to the user's soundcollection microphone). The audio signals determined to be received froma source different from the imaged individuals may be amplifieddifferently (e.g., using a higher gain) relative to audio signalsreceived from imaged individuals. In an embodiment, the system maydetect the presence of a speakerphone participant, at least in part,through detection in a captured image of a speakerphone device orsimilar device present in the FOV of the system camera.

In some embodiments, the system may automatically identify individualspresent in a group discussion (e.g., via facial recognition, voicerecognition, or both) and record the discussion participants in adatabase. The system may also determine the identity of at least oneperson participating in the discussion by phone (or outside the cameraFOV) through voice recognition or based on other criteria, such asmeeting invite records, prior known associations, etc. The system mayrecord the identities of the participants. In some embodiments, thesystem may amplify the voice of a person that previously appeared in acamera FOV but has exited the FOV, and later speaks (e.g., during a carride, or in a user's home, etc.). In some embodiments, the system mayalso amplify certain sound signals. For example, the system may amplifya fire alarm, a siren, crying by a kid, a voice warning (e.g.,“Mayday!”). Alternatively or additionally, some (predetermined,recognized or not) sounds, whether recognized or not, or predeterminedmay be amplified and transmitted at a delay. For example, in an airportwhen there is an announcement about a flight, the system may realizethat this is an important announcement only after the flight number ismentioned. The system may play the whole announcement even though thatvoice is not of anyone known to the user.

FIG. 29 illustrates an exemplary hearing aid system. User 2901 may weara wearable device 2931. Wearable device 2931 may include an image sensorconfigured to capture images of the environment of user 2901. Asillustrated in FIG. 29, user 2901 may sit by one side of a table. Afirst individual 2911 and a second individual 2912 may sit by anotherside of the table. The image sensor of wearable device 2931 may captureone or more images of the environment of user 2901, including firstindividual 2911, second individual 2912, and a speakerphone 2921.

FIGS. 30A and 30B illustrate exemplary images 3000A and 3000B of theenvironment of user 2901 illustrated in FIG. 29. Image 3000A may includea representation 3011 of first individual 3011, a representation 3012 ofsecond individual 3012, and a representation 3021 of speakerphone 2921.Image 3000B may include representation 3012 of second individual 3012and representation 3021 of speakerphone 2921 (the first individual maybe out of the FOV of the camera). Wearable device 2931 may also includeat least one processor configured to analyze the images captured by theimage sensor. The processor may also identify a representation of one ormore individuals and one or more objects included in the images, basedon the image analysis. For example, the processor may receive image3000A and/or 3000B (illustrated in FIGS. 30A and 30B) from the imagesensor and identify representations of first individual 2911, secondindividual 2912, and speakerphone 2921 included in the image. In someembodiments, the processor may be programmed to perform one or moresteps of process 3110, process 3130, and/or process 3150 (illustrated inFIGS. 31A, 31B, and 31C, respectively).

In some embodiments, wearable device 2931 may be configured toautomatically identify one or more individuals, based on the images, theaudio signal(s) detected, another type of data, or the like, or acombination thereof. For instance, wearable device 2931 mayautomatically identify first individual 2911 and second individual 2912based on the images using facial recognition technologies. Alternativelyor additionally, wearable device 2931 may automatically identify anindividual based on voice recognition (e.g., the voice print of theindividual) associated with an audio signal detected. For instance,wearable device 2931 may automatically identify an individual who is notin the room with user 2901 and is participating in a conference call viaspeakerphone 2921, based on a detected audio signal associated with theindividual. Alternatively or additionally, wearable device 2931 mayautomatically identify an individual based on a calendar inviteassociated with the user or prior known associations of the user. Forexample, wearable device 2931 may receive data relating to a calendarinvite, which may include the identity of one or more participants.Wearable device 2931 may identify an individual as one of theparticipants included in the calendar invite. In some embodiments,wearable device 2931 may further record the identification of the one ormore individuals in a database.

Wearable device 2931 may further include at least one microphoneconfigured to receive one or more audio signals from the environment ofuser 2901. For example, the microphone may be configured to receive (ordetect) an audio signal associated with the first individual 2911 and/orthe second individual 2912 and/or additional audio such as backgroundnoise. The microphone may also be configured to receive (or detect) anaudio signal associated with the speakerphone 2921 (e.g., the voice of athird individual participating in the conference through thespeakerphone 2921).

In some embodiments, the microphone may include a directional microphone(e.g., a bi-directional microphone, an omnidirectional microphone,etc.), a microphone array, or the like, or a combination thereof. Insome embodiments, the processor may be configured to determine the speakdirection of each of one or more identified individuals and/or the user,based on the audio signals received. For example, the microphone mayinclude one or more directional microphones, and the processor may beconfigured to determine the speak direction of an individual based onthe audio signal associated with the individual.

Wearable device 2931 may also determine, based on analysis of theimages, whether an audio signal received is associated with a voice ofthe one or more individuals detected in the images. For example,wearable device 2931 may receive a first audio signal and determine thatthe first audio signal is not associated with any of the individuals(e.g., first individual 2911 and second individual 2912) identified inthe images based on analysis of the images. Additionally, wearabledevice 2931 may receive a second audio signal and determine that thesecond audio signal is associated with a voice of first individual 2911.Wearable device 2931 may further determine the sources of the audiosignals based on the images and/or the audio signals. For example,wearable device 2931 may detect lip movements associated with firstindividual 2911 based on the image analysis. Wearable device 2931 mayalso determine that the detected lip movements correspond to the secondaudio signal and determine the source of the second audio signal to befirst individual 2911. As another example, wearable device 2931 maydetermine that an audio signal originated from a speaker. In someembodiments, the speaker may include a speakerphone, a network-connectedspeaker (e.g., Bluetooth or WiFi speaker), a wired speaker, a cellphone, or the like, or a combination thereof. By way of example,wearable device 2931 may determine that the speaker is included in aspeakerphone by detecting, through analysis of one or more of theimages, a representation of a device recognized as a speakerphone.

Wearable device 2931 may further cause a first amplification of thefirst audio signal and a second amplification of the second audiosignal. The first amplification may differ from the second amplificationin at least one aspect. For example, wearable device 2931 may amplifythe first audio signal by a first gain level and amplify the secondaudio signal by a second gain level. In some embodiments, the first gainlevel may be greater than the second gain level.

Wearable device 2931 may be in communication with a hearing interfacedevice (e.g., an earphone) configured to receive audio signals andprovide sound to an ear of user 2901. For example, wearable device 2931may cause transmission of at least one of the first audio signal,amplified according to the first amplification, and the second audiosignal, amplified according to the second amplification, to a hearinginterface device configured to provide sound to an ear of user 2901. Forexample, the processor may cause a transmitter to transmit at least oneof the first audio signal, amplified according to the firstamplification, and the second audio signal, amplified according to thesecond amplification to a hearing interface device via a wirelessnetwork (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or via near-fieldcapacitive coupling, other short-range wireless techniques, or via awired connection.

In some embodiments, a hearing interface device may include a speakerassociated with an earpiece. For example, the hearing interface devicemay include an in-the-ear, in-the-canal, completely-in-canal,behind-the-ear, on-the-ear, receiver-in-canal, open fit, or variousother styles of earphones. As another example, the hearing interfacedevice may include a speaker included in a wearable device (e.g.,wearable device 2631). In some embodiments, the hearing interface devicemay include an earphone, a headphone, a speaker, or the like, or acombination thereof. In some embodiments, the hearing interface devicemay include a bone conduction microphone.

In some embodiments, the microphone and the image sensor (or thewearable camera that includes the image sensor) may be included in acommon housing. For example, wearable device 2931 illustrated in FIG. 29may include both the microphone and the wearable camera in a commonhousing. Alternatively, the microphone may be included in a housingdifferent from a housing in which the wearable camera is installed.

In some embodiments, the processor may be included in a common housingwith at least one of the microphone and the wearable camera. Forexample, the processor may also be included in a common housing in whichboth the microphone and the wearable camera are included. Alternatively,the processor may be included in a separate housing from a commonhousing where the microphone and the wearable camera are installed. Theprocessor may also be configured to receive data (e.g., the capturedimages, the detected audio signals, etc.) from the wearable cameraand/or the microphone via a wireless link between a transmitter in thecommon housing (in which the microphone and the wearable camera areincluded) and receiver in the second housing (in which the processor isincluded).

FIG. 31A is a flowchart of an exemplary process for selectivelyamplifying audio signals. At step 3111, the hearing aid system mayreceive a plurality of images captured by a camera. For example, thehearing aid system may include a processor (e.g., processor 210)configured to receive images of the environment of the user captured byan image sensor (e.g., image sensor 220). In some embodiments, the imagesensor may be part of a camera included the hearing aid system. By wayof example, as illustrated in FIG. 29, user 2901 may wear a wearabledevice 2931 that may include an image sensor configured to captureimages of the environment of the user. The processor of the hearing aidsystem may receive the images from wearable device 2931.

In some embodiments, the hearing aid system may control the image sensorto capture images. For example, the processor may detect a gestureperformed by the user (a finger-pointing gesture) and control the imagesensor to capture images based on the detected gesture (e.g., adjustingthe field of view of the image sensor based on the direction of thefinger-pointing gesture). As another example, the hearing aid system mayinclude a microphone configured to detect (or receive) audio signalsfrom the environment of the user. The processor may receive the audiosignals from the microphone and detect a voice by one or moreindividuals nearby. The processor may control the image sensor tocapture images if a voice is detected.

In some embodiments, the processor may receive data from or transmitdata to the image sensor over one or more networks via any knownwireless standard (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or vianear-field capacitive coupling, other short-range wireless techniques,or via a wired connection. For example, the processor may also beconfigured to receive data (e.g., the captured images, etc.) from theimage sensor via a wireless link between a transmitter in a housing inwhich the image sensor is included and a receiver in a housing in whichthe processor is included.

At step 3113, the hearing aid system may identify a representation ofone or more individuals in the plurality of images. For example, theprocessor may identify a representation of first individual 2911 andsecond individual 2912 in the images. For instance, the processor mayanalyze image 3000A illustrated in FIG. 30A and identify representation3011 of first individual 2911 and representation 3012 of secondindividual 2912. In some embodiments, the processor may also identify arepresentation of one or more objects included in the images. Forexample, the processor may identify in image 3000A representation 3021of speakerphone 2921 (illustrated in FIG. 29).

In some embodiments, the processor may be configured to automaticallyidentify the one or more individuals, based on the images, the audiosignal(s) detected, another type of data, or the like, or a combinationthereof. For instance, the processor may automatically identify firstindividual 2911 and second individual 2912 based on the images usingfacial recognition technologies. Alternatively or additionally, theprocessor may automatically identify an individual based on voicerecognition (e.g., the voice print of the individual) associated with anaudio signal detected. Alternatively or additionally, the processor mayautomatically identify an individual based on a calendar inviteassociated with the user or prior known associations of the user. Forexample, the processor may receive data relating to a calendar invite,which may include the identity of one or more participants. Theprocessor may identify first individual 2911 as one of the participantsincluded in the calendar invite. In some embodiments, the processor mayfurther record the identification of the one or more individuals in adatabase.

At step 3115, the hearing aid system may receive from the at least onemicrophone a first audio signal associated with a voice. For example,the hearing aid system may include a microphone configured to receive(or detect) audio signals from the environment of user 2901, including afirst audio signal associated with a voice.

At step 3117, the hearing aid system may determine, based on analysis ofthe plurality of images, that the first audio signal is not associatedwith a voice of any of the one or more individuals. For example, theprocessor may analyze the images to detect the facial expression (e.g.,lip movements) of the individual(s) detected in the images. Theprocessor may determine the first audio signal is not associated with avoice of any of the one or more individuals by analyzing detected lipmovements associated with mouths of first individual 2911 and/or secondindividual 2912, and determine that the first audio signal does notcorrespond to the detected lip movements associated with mouths of firstindividual 2911 and/or second individual 2912. The first audio signalmay be associated with a voice of an individual who is outside of theFOV of the camera (e.g., an individual participating in a conferencecall through speakerphone 2921 or an individual who is in the room butsits far away from user 2901).

In some embodiments, the processor may identify the source of the firstaudio signal. For example, the processor may automatically identify anindividual based on voice recognition (e.g., the voice print of theindividual) associated with an audio signal detected. By way of example,the processor may automatically identify an individual who is not in theroom with user 2901 and is participating a conference call viaspeakerphone 2921, based on the analysis of the first audio signal.Alternatively or additionally, the processor may automatically identifyan individual based on a calendar invite associated with the user orprior known associations of the user. For example, the processor mayreceive data relating to a calendar invite, which may include theidentify of one or more participants. The processor may identify anindividual as one of the participants included in the calendar invite.In some embodiments, the processor may further record the identificationof the one or more individuals in a database. The participant may becaptured earlier and then disappeared from the image (maybe went toanother room)

At step 3119, the hearing aid system may receive from the at least onemicrophone a second audio signal associated with a voice. The hearingaid system may receive the first audio signal and the second audiosignal at the same time or at different times. In some embodiments, atleast portion of the first audio signal may overlap with a portion ofthe second audio signal. The voices may be separate by any voiceseparation technique, for example using periods in which only onespeaker speaks, as detailed above.

At step 3121, the hearing aid system may determine, based on analysis ofthe plurality of images, that the second audio signal is associated witha voice of one of the one or more individuals. For example, theprocessor may determine the second audio signal is associated with firstindividual 2911 (and/or second individual 2912) by analyzing detectedlip movements associated with mouths of first individual 2911 (and/orsecond individual 2912) and determining that the second audio signalcorresponds to the detected lip movements associated with a mouth offirst individual 2911 (and/or second individual 2912).

In some embodiments, the processor may be configured to automaticallyidentify one or more individuals associated with the first audio signaland/or the second audio signal, based on the images, the audio signal(s)detected, another type of data, or the like, or a combination thereof.For instance, the processor may automatically identify first individual2911 (and/or second individual 2912) who is associated with the secondaudio signal, based on the images using facial recognition technologies.Alternatively or additionally, the processor may automatically identifyan individual based on voice recognition (e.g., the voice print of theindividual) associated with an audio signal detected. Alternatively oradditionally, the processor may automatically identify an individualbased on a calendar invite associated with the user or prior knownassociations of the user. For example, the processor may receive datarelating to a calendar invite, which may include the identity of one ormore participants. The processor may identify an individual as one ofthe participants included in the calendar invite. In some embodiments,the processor may further record the identification of the one or moreindividuals in a database.

At step 3123, the hearing aid system may cause a first amplification ofthe first audio signal and a second amplification of the second audiosignal. The first amplification may differ from the second amplificationin at least one aspect. For example, the processor may amplify the firstaudio signal by a first gain level and amplify the second audio signalby a second gain level. In some embodiments, the first gain level may begreater than the second gain level.

In some embodiments, the processor may amplify the first audio signal toa first predetermined sound level and the second audio signal to asecond predetermined sound level. The first predetermined sound levelmay be lower than, greater than, or the same as the second predeterminedsound level. Alternatively or additionally, the processor may amplifythe first audio signal by increasing the sound level by a firstpercentage and amplify the second audio signal by increasing the soundlevel by a second percentage.

At step 3125, the hearing aid system may cause transmission of at leastone of the first audio signal, amplified according to the firstamplification, and the second audio signal, amplified according to thesecond amplification, to a hearing interface device configured toprovide sound to an ear of the user. For example, the processor mayinclude a transmitter configured to transmit the amplified audiosignal(s) (e.g., amplified first audio signal, amplified second audiosignal, etc.) to a hearing interface device via a wireless network(e.g., cellular, Wi-Fi, Bluetooth®, etc.), or via near-field capacitivecoupling, other short-range wireless techniques, or via a wiredconnection. A hearing interface device may include a speaker associatedwith an earpiece. For example, the hearing interface device may includean in-ear earphone. As another example, the hearing interface device mayinclude a speaker included in a wearable device (e.g., wearable device2931). In some embodiments, the hearing interface device may include anearphone, a headphone, a speaker, or the like, or a combination thereof.

FIG. 31B is a flowchart of an exemplary process 3130 for selectivelyamplifying audio signals. At step 3131, the hearing aid system (e.g.,apparatus 110) may receive a first plurality of images. For example, thehearing aid system may include a processor (e.g., processor 210)configured to receive images of the environment of the user captured byan image sensor (e.g., image sensor 220). In some embodiments, the imagesensor may be part of a camera included in the hearing aid system. Byway of example, as illustrated in FIG. 29, user 2901 may wear theprocessor that may include an image sensor configured to capture imagesof the environment of the user. The processor of the hearing aid systemmay receive the images from wearable device 2931.

At step 3133, the hearing aid system may identify a representation of anindividual in the first plurality of images. In some embodiments, thehearing aid system may identify a representation of an individual (andor an object) in the first plurality of images using a method similar tothat of step 3113 of process 3110 described above. For example, theprocessor may be configured to analyze image 3000A and identifyrepresentation 3011 of first individual 2911 and/or representation 3012of second individual 2912 in image 3000A based on the image analysis. Insome embodiments, the processor may also automatically identify theindividual and record the identification of the individual into adatabase as described elsewhere in this disclosure.

At step 3135, the hearing aid system receive from the at least onemicrophone a first audio signal representative of a voice. For example,the hearing aid system may include a microphone configured to receive(or detect) audio signals from the environment of user 2901, including afirst audio signal associated with a voice. In some embodiments, themicrophone may include a directional microphone (e.g., a bi-directionalmicrophone, an omnidirectional microphone, etc.), a microphone array, orthe like, or a combination thereof.

In some embodiments, the microphone and the image sensor (or thewearable camera that includes the image sensor) may be included in acommon housing. Alternatively, the microphone may be included in ahousing different from a housing in which the wearable camera isinstalled. In some embodiments, the processor may be included in acommon housing with at least one of the microphone and the wearablecamera. For example, the processor may also be included in a commonhousing in which both the microphone and the wearable camera areincluded. Alternatively, the processor may be included in a separatehousing from a common housing where the microphone and the wearablecamera are installed. The processor may also be configured to receivedata (e.g., the captured images, the detected audio signals, etc.) fromthe wearable camera and/or the microphone via a wireless link between atransmitter in the common housing (in which the microphone and thewearable camera are included) and receiver in the second housing (inwhich the processor is included).

At step 3137, the hearing aid system may determine, based on analysis ofthe first plurality of images, that the first audio signalrepresentative of a voice is associated with the individual. In someembodiments, the hearing aid system may determine, based on analysis ofthe first plurality of images, that the first audio signalrepresentative of a voice is associated with the individual using amethod similar to that of step 3121 of process 3110 described above. Forexample, the processor may determine the first audio signal isassociated with first individual 2911 (and/or second individual 2912) byanalyzing detected lip movements associated with mouths of firstindividual 2911 (and/or second individual 2912) and determining that thefirst audio signal corresponds to the detected lip movements associatedwith a mouth of first individual 2911 (and/or second individual 2912).

At step 3139, the hearing aid system may selectively amplify the firstaudio signal over other audio signals received from the at least onemicrophone representative of sounds from sources other than theindividual. For example, the processor may amplify the first audiosignal by a first gain level. Alternatively or additionally, theprocessor may amplify the first audio signal to a first predeterminedsound level. Alternatively or additionally, the processor may amplifythe first audio signal by increasing the sound level by a percentage.

At step 3141, the hearing aid system may receive a second plurality ofimages captured by the camera. In some embodiments, the hearing aidsystem may receive the second plurality of images after the firstplurality of images. For example, the processor may receive the firstplurality of images captured by the camera during a first period of timeand receive the second plurality of images captured by the camera duringa second period of time. The hearing aid system may receive the secondplurality of images using a method similar to step 3131 described above.For example, the hearing aid system may receive image 3000B illustratedin FIG. 30B (as one of the second plurality of images).

At step 3143, the hearing aid system may receive from the at least onemicrophone a second audio signal representative of a voice associatedwith the individual. In some embodiments, the hearing aid system mayreceive the second audio signal representative of a voice after thefirst audio signal. For example, the second audio signal may be receivedfrom speaker (e.g., speakerphone 2921) through which the individualspeaks (e.g., the individual is speaking via speakerphone 2921 in atelephonic call). As another example, the second audio signal may bereceived from the individual directly. In some embodiments, the hearingaid system may receive the second audio signal using a method similar tostep 3135 described above.

At step 3145, the hearing aid system may determine, based on analysis ofthe second plurality of images, that the individual is not representedin the second plurality of images. For example, the hearing aid systemmay determine, based on analysis of image 3000B illustrated in FIG. 30B,first individual 2911 may be outside of the FOV of the camera when image3000B is captured (e.g., having left the room or being outside of theFOV despite remaining in the room). The processor may analyze the secondplurality of images and determine that the individual is not representedin the second plurality of images.

At step 3147, the hearing aid system may selectively amplify the secondaudio signal over other received audio signals representative of soundsfrom sources other than the individual. In some embodiments, theprocessor may selectively amplify the second audio signal by a secondgain level. Alternatively or add-itionally, the processor may amplifythe second audio signal to a second predetermined sound level.Alternatively or additionally, the processor may amplify the secondaudio signal by increasing the sound level by a percentage.

At step 3149, the hearing aid system may cause transmission of at leastone of the selectively amplified first audio signal or the selectivelyamplified second audio signal to a hearing interface device.Transmission of an amplified audio signal to the hearing interfacedevice is described elsewhere in this disclosure. For example, theprocessor may cause a transmitter to transmit at least one of theselectively amplified first audio signal or the selectively amplifiedsecond audio signal to the hearing interface device via a wirelessnetwork (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or via near-fieldcapacitive coupling, other short-range wireless techniques, or via awired connection.

In some embodiments, the hearing interface device may include a speakerassociated with an earpiece. For example, the hearing interface devicemay include an in-ear earphone. As another example, the hearinginterface device may include a speaker included in a wearable device(e.g., wearable device 2631). In some embodiments, the hearing interfacedevice may include an earphone, a headphone, a speaker, or the like, ora combination thereof. In some embodiments, the hearing interface devicemay include a bone conduction microphone.

FIG. 31C is a flowchart of an exemplary process for selectivelyamplifying audio signals. At step 3151, the hearing aid system mayreceive a plurality of images. The hearing aid system may receive aplurality of images based on a method similar to step 3111 of process3110 described above. For example, user 2901 may wear wearable device2931 that may include an image sensor configured to capture images ofthe environment of the user. The processor of the hearing aid system mayreceive the images from wearable device 2931.

At step 3153, the hearing aid system may identify a representation ofone or more individuals in the plurality of images. The hearing aidsystem may identify a representation of one or more individuals in theplurality of images based on a method similar to step 3113 of process3110 described above. For instance, the processor may analyze image3000A illustrated in FIG. 30A, and identify representation 3011 of firstindividual 2911 and representation 3012 of second individual 2912. Insome embodiments, the processor may also identify a representation ofone or more objects included in the images. For example, the processormay identify in image 3000A representation 3021 of speakerphone 2921(illustrated in FIG. 29).

At step 3155, the hearing aid system may receive from the at least onemicrophone a first audio signal associated with a voice. In someembodiments, the hearing aid system may receive from the at least onemicrophone a first audio signal based on a method similar to step 3115of process 3110 described above. For example, the processor may includea microphone configured to receive (or detect) audio signals from theenvironment of user 2901, including a first audio signal associated witha voice.

At step 3157, the hearing aid system may determine, based on analysis ofthe images, that the first audio signal is not associated with a voiceof any of the one or more individuals. In some embodiments, thedetermination may be based on a method similar to step 3117 of process3110 described above. For example, the processor may analyze the imagesto detect the facial expression (e.g., lip movements) of theindividual(s) detected in the first plurality of images. The processormay determine the first audio signal is not associated with a voice ofany of the one or more individuals is made by analyzing detected lipmovements associated with mouths of first individual 2911 and/or secondindividual 2912, and determine that the first audio signal does notcorrespond to the detected lip movements associated with mouths of firstindividual 2911 and/or second individual 2912.

At step 3159, the hearing aid system may determine, based on analysis ofthe audio signal, that the audio signal is associated with at least oneindicator that the audio signal is related to a public announcement. Forexample, the processor may analyze the received audio signal todetermine the content associated with the audio signal. The processormay also determine that the audio signal is related to a publicannouncement based on the content. A public announcement may include acommunication intended for a group of people, of which the device usermay be a part (e.g., a gate announcement broadcasted at an airport). Asanother example, a public announcement may be a call for help (e.g.,Mayday, etc.).

In some embodiments, the at least one indicator that the audio signal isrelated to a public announcement may include a recognized sound, word orphrase associated with the audio signal. For example, the processor mayrecognize one or more words or phrases that are associated with anairport announcement (e.g., a flight number), and determine that theaudio signal is related to a public announcement based on the recognizedword or phrase. As another example, the audio signal may include a word(or phrase) such as “help,” “watch out,” “attention,” “announcement” (orsimilar words or phrases in other languages), or the like, or acombination thereof. The processor may analyze the audio signal andrecognize such word (or phrase) and determine that the audio signal isrelated to a public announcement based on the recognized word (orphrase). Alternatively or additionally, the at least one indicator thatthe audio signal is related to a public announcement may include avolume level of the audio signal relative to an ambient noise level,which may indicate that the audio signal relates to a yell, scream, apublic announcement over a loudspeaker, or the like, or a combinationthereof. For example, the processor may determine that the volume levelof the audio signal is greater than the ambient noise level by athreshold, and determine that the audio signal may be related to apublic announcement or an event that needs attention. Alternatively oradditionally, the at least one indicator that the audio signal isrelated to a public announcement includes at least one signal componentassociated with the audio signal indicative of production of the audiosignal by a loudspeaker. For example, the audio signal may be related toa broadcast over one or more loudspeakers, which may include one or moresignal characteristics indicating amplification of the voice orreproduction of the voice over a public address system.

At step 3161, the hearing aid system may cause selective amplificationof the audio signal based on the determination that the audio signal isassociated with at least one indicator that the audio signal relates toa public announcement. For example, the processor may amplify the audiosignal associated with a public announcement. In some embodiments, theprocessor may amplify the audio signal to a predetermined sound level.Alternatively or additionally, the processor may amplify the audiosignal by increasing the sound level by a percentage. Alternatively oradditionally, while amplifying an audio signal, the processor may beconfigured to attenuating one or more other audio signals (by, forexample, decreasing the sound level of the other signal(s) to apredetermined sound level or by a predetermined percentage). Forexample, the processor may be configured to amplify the audio signalassociated with a public announcement by 50% and attenuate one or moreother audio signals by 50%.

In some embodiments, the processor may determine whether the audiosignal relating to a public announcement (e.g., a public announcement atan airport) is relevant to the user and may selectively amplify theaudio signal based on the result of the determination. For example, theprocessor may determine that the audio signal relates to a flight thatis irrelevant to the user and may not amplify the audio signalaccordingly. As another example, the processor may determine that thepublic announcement associated with the audio signal relates to a gatechange of the user's flight. The processor may also selectively amplifythe audio signal. In some embodiments, the processor may determinewhether airport announcement is relevant to the user based on automaticreview of a calendar entry or reservation notice stored on a mobiledevice associated with the user. For example, the processor may accessthe data relating to a calendar entry or reservation notice stored on amobile device associated with the user and determine that the flightinformation (e.g., the flight number) relating to the flight that theuser is going to take based on the accessed data. The processor may alsodetermine whether a public announcement associated with the audio signalis relevant to the user based on the flight information and the audiosignal (e.g., the content of the message associated with the audiosignal).

In some embodiments, the hearing aid system may capture one or moreaudio signals received during a moving time window of a predeterminedlength, and the processor may be programmed to cause selectiveamplification and transmission of a portion of the audio signal receivedwithin the moving time window but prior to the determination that theaudio signal is related to a public announcement. For example, thehearing aid system may recognize that a stream of voice communications(in the form of one or more audio signals) includes a publicannouncement. The recognition may be after the announcement begins. Theprocessor may use a moving time window of the captured audio signal(s)to go back to the beginning of the public announcement and extractinformation relating to the full announcement from within the timewindow and selectively amplify that full announcement (in the form ofone or more audio signals) for the user. The amplified audio signal(s)may be transmitted to the user time-delayed relative to the originalannouncement.

At step 3163, the hearing aid system may cause transmission of theselectively amplified audio signal to a hearing interface device. Forexample, the processor may cause a transmitter to transmit the amplifiedaudio signal to a hearing interface device via a wireless network (e.g.,cellular, Wi-Fi, Bluetooth®, etc.), or via near-field capacitivecoupling, other short-range wireless techniques, or via a wiredconnection. Alternatively or additionally, the processor may causetransmission of the unprocessed audio signal(s) (and/or the selectivelyattenuated audio signal) to the hearing interface device.

Selectively Conditioning Audio Signals

In accordance with various embodiments of the disclosure, a wearableapparatus, such as apparatus 110, may be configured to use audioinformation in addition to image information. For example, apparatus 110may detect and capture sounds in an environment of the user (e.g., user100), via one or more microphones. Apparatus 110 may use this audioinformation instead of, or in combination with, image and/or videoinformation to determine situations, identify persons, performactivities, or the like. The image and/or video information maysupplement the audio information in various situations for peoplewearing a hearing aid system. For example, people that use a hearing aidoften find that the hearing aid does not perform optimally in a crowdedenvironment. In such cases, various environmental sounds may beamplified and impede a user who wears the hearing aid (e.g., user 100)from clearly distinguishing sounds that are directly relevant to user100, such as conversational words or sounds from a person communicatingwith user 100. In such cases, image data may be used to identify anindividual relevant to user 100 (e.g., an individual in conversationwith user 100).

In accordance with an embodiment of the disclosure, a hearing aid systemis provided. The hearing aid system may include a wearable cameraconfigured to capture a plurality of images from an environment of user100. In various embodiments, the hearing aid system may include at leastone microphone configured to capture sounds from an environment of theuser. In some embodiments, the hearing aid system may include more thanone microphone. In an example embodiment, the hearing aid system mayinclude a first microphone for capturing audio signals in a firstwavelength range and a second microphone for capturing audio signals ina second wavelength range.

The hearing aid system may include at least one processor programmed toreceive the plurality of images captured by the wearable camera andidentify a representation of at least one individual in at least one ofthe plurality of images. The processor may be configured to use acomputer-based model to extract an image of a person from the receivedimage. For example, the processor may use a neural network model (e.g.,a convolutional neural network (CNN)), to recognize an image of a personin the received image. In an example embodiment, the hearing aid systemmay be configured to capture an image in a direction normal to a face ofuser 100 (e.g., determined based on a direction associated with a chinof the user, which direction can be normal to the chin of user 100) todiscern a speaker in the captured image.

The hearing aid system may be incorporated into apparatus 110, or insome embodiments, apparatus 110 may constitute the hearing aid system.As previously described, in connection with FIG. 2, apparatus 110 maytransfer or receive data to/from server 250 via network 240. In anexample embodiment, an image of a person in the received image may betransferred to server 250 for analysis of the image. Server 250 mayinclude a processor configured to access a database associated withserver 250 containing various images of people that are related to user100 and compare these images with the one or more images of a persontransferred to server 250 by apparatus 110.

In an example embodiment, the database of server 250 may select imagesof friends of user 100, relatives of user 100, co-workers of user 100,persons whom user 100 has encountered in the past, and the like forcomparison with the one or more images of a person captured by thehearing aid system. In some embodiments, the hearing aid system mayaccess to a global positioning system (GPS) and may determine thelocation of user 100. For example, the hearing aid system may include aGPS system, or it may communicate with a mobile device (e.g.,smartphone, tablet, laptop, etc.) of user 100 that includes a GPS system(or alternative system for determining position of the mobile device,such as Wi-Fi, local network, etc.) to obtain location data (e.g.,coordinates of user 100, address of user 100, IP address of the mobiledevice of user 100, etc.). The hearing aid system may communicate thelocation of user 100 to server 250. In an example embodiment, server 250may be configured to select from a database (e.g., stored in server 250)images of people who are likely to be found at the location of user 100.For example, when user 100 is located at a work site, images ofco-workers may be selected first.

Additionally, or alternatively, the hearing aid system may communicateto server 250 a time when the images of an environment of user 100 werecaptured by the wearable camera. In an example embodiment, server 250may be configured to select from a database (e.g., stored in server 250)images of people who are likely to be found at the location of user 100at the communicated time. For example, when user 100 is located at home,and the time corresponds to dinner time, the images of relatives may beselected first.

In various embodiments, the image of a person obtained from capturedimages by the wearable camera of the hearing aid system may be comparedby a processor of server 250 with various images selected from thedatabase of server 250 using any suitable approaches. For example,images may be compared by using neural networks such as CNN, or anyother suitable computer-based methods. In some embodiments, acomputer-based model may assign a likelihood indicating to what degreethe image of the person obtained from captured images matches with atleast one image found in the database of server 250. In an exampleembodiment, the likelihood may be a probability of the image of theperson obtained from captured images matching with at least one imagefound in the database of server 250 and may be in the range of valuesfrom zero to one.

In various embodiments, an image in the database of server 250 may havean associated data record that can be stored in the database of server250 in association with the related image. For example, the image fromthe database may have a data record associated with a person, and thedata record may include a person's name, a relationship to user 100,dates and times the person met user 100, and the like. In some cases,one data record may be associated with multiple images located in thedatabase of server 250. A data record may be retrieved from the databasefor one or more associated images. For example, server 250 may beconfigured to retrieve a data record for the one or more associatedimages using a processor. Additionally, or alternatively, one or moreimages may be retrieved from the database for the associated datarecord. In an example embodiment, the image of a person obtained fromcaptured images may be compared with more than one image from thedatabase of server 250 that corresponds to the same data record toestablish the likelihood. In various embodiments, if the likelihood isabove a predetermined threshold value, the hearing aid system mayestablish that the image of the person obtained from captured imagesmatches the data record from the database.

In an example embodiment, data records for images stored in the databaseof server 250 may be linked. For example, a data record for one personmay be linked with a data record for another person, where a link mayinclude any suitable relationship information between the linked datarecords. In some cases, the link may be used to define the relationshipbetween people whose data records are stored in the database. Forexample, the people may be defined as co-workers, friends, competitors,neighbors, teammates, admirers of the same product, person, singer,actor, and the like. In various embodiments, server 250 may use linksbetween data records to re-evaluate the likelihood that the personidentified from captured images matches the image of an individual foundin the database of server 250. For example, if a data record for animage of the individual found in the database of server 250 includes alink to a data record of user 100, the likelihood value may beincreased. For example, if the link indicates that user 100 is aco-worker of the individual, and user 100 is located at a work site, thelikelihood value may be increased. In some embodiments, a firstencounter with a first individual (e.g., a coworker) may affect thelikelihood value that a second individual (e.g., another coworker)identified from captured images during a second encounter, matches adata record for an individual found in the database of server 250.

While the discussion above describes using server 250 for analyzingimages captured by a wearable device of user 100, additionally, oralternatively, a processor of the hearing aid system may be used foranalyzing the images. For example, the processor of the hearing aidsystem may be configured to receive various images or characteristics ofpersons from the database of server 250 as well as the associated datarecords for these images or characteristics, and compare the receivedimages or characteristics with an image or characteristics of the personidentified in the captured images. Similar to the embodiments discussedabove, the processor of the hearing aid system may use a computer-basedmodel to compare images and may receive images from the database thatare relevant to location or time for user 100. In an example embodiment,the computer-based model may include a neural network such as aconvolutional neural network (CNN). In some embodiments, thedetermination of whether the at least one individual is a recognizedindividual may be based on an output of a trained neural networksupplied with the at least one of the plurality of images that can beused to analyze one or more images. In some embodiments, thedetermination of whether the at least one individual is a recognizedindividual may be based on one or more facial features associated withthe at least one individual that are detected based on analysis of theat least one of the plurality of images. For example, a computer-basedmodel such as CNN may be used to analyze images and compare facialfeatures or relations between facial features of the person identifiedin the captured images with facial features or relations therebetween ofpeople found in images stored in the database of server 250. In someembodiments, a video of person's facial dynamic movements may becompared with video data record for various people, obtained from thedatabase, in order to establish that the person captured in the video isa recognized individual.

In various embodiments, the hearing aid system may include at least oneprocessor programmed to receive an audio signal from the at least onemicrophone. In an example embodiment, the at least one processor may beconfigured to use a computer-based model to determine whether thereceived audio signal is associated with a recognized individual. Forexample, the computer-based model may be a neural network model (e.g.,convolutional neural network (CNN)), and the like. In some cases, theaudio signal may include multiple audio signals from multiple sources(e.g., an audio signal from a speaker in conversation with user 100, anenvironmental audio signal, and the like). In various embodiments, thedetermination of whether the at least one individual is a recognizedindividual may be based on analysis of the at least one audio signal(e.g., an audio signal related to user 100 conversing with one or morespeakers) received by the microphone of the hearing aid system. In anexample embodiment, the audio signal may be determined to be associatedwith the recognized individual based on a detected look direction forthe user, determined based on a direction associated with a chin of theuser detected in the at least one of the plurality of images.

In an example embodiment, a detection in the audio signal of one or moreof predetermined voice characteristics associated with one or morerecognized individuals (e.g., individuals whose voiceprints and datarecords are available, for example, in the database of server 250, orelsewhere) may be used to identify and recognize one or more speakers.For example, the detection of voice characteristics of a recognizedindividual may determine whether the received audio signal is associatedwith the recognized individual. As used herein, the term “voiceprint”may refer to a set of measurable features (or feature ranges) of a humanvoice that uniquely identifies a speaker. In some embodiments, theseparameters may based on the physical configuration of a speaker's mouth,throat, and additional organs, and/or may be expressed as a set ofsounds related to various syllables pronounced by the speaker, a set ofsounds related to various words pronounced by the speaker, a modulationor inflection of a voice of the speaker, cadence of a speech of thespeaker, and the like.

In various embodiments, server 250 may receive images and audioinformation (e.g., voiceprints) for various individuals from a varietyof sources. For example, FIG. 32 shows server 250 receiving images 3211and audio data 3212 from user 100 wearing apparatus 110. In some cases,images and audio data may be submitted to server 250 via computingdevice 120 (e.g., a smartphone, laptop, tablet, and the like). In someembodiments, server 250 may be configured to access information (e.g.,images, video, audio data, etc.) available over a social network 3220(e.g., a Facebook® page/LinkedIn® page, email, Instagram®, and the like)associated either with user 100 or with one or more individualsidentified in images 3211, as shown in FIG. 32. The information fromsocial network 3220 may include data related to friends of user 100, tofriends of friends of user 100, and the like. In some embodiments,server 250 may receive information from individuals that do not use thehearing aid system (e.g., apparatus 110, as shown in FIG. 32) but whomay have a user profile associated with server 250. For example, user3230 may be a relative, co-worker, friend, and the like of user 100, andmay have a user profile associated with server 250. In variousembodiments, user 3230 may take images/video and/or audio data 3231(e.g., a selfie as shown in FIG. 32) and upload data 3231 to server 250.In various embodiments, user 3230 may upload information to server 250such as an associated data record (e.g., a name of user 3230, alocation, and the like). In an example embodiment, the one or moreprocessors may be programmed to transmit images 3211 and audio data 3212to the database relating to encounters with individuals. For example,the one or more processors may be configured to transmit images 3211 andaudio data 3212 when a speaker for a conversation is identified andrecognized, or/and images 3211 and audio data 3212 related to variousconversations with various speakers even if those speakers are notrecognized.

In various embodiments, the hearing aid system may be configured tointeract with user 100 via visual or audio data. For example, thehearing aid system may interact with user 100 via a display using audiosignals delivered to user 100 via earpiece devices, and the like. In anexample embodiment, the hearing aid system may determine whether the atleast one individual is a recognized individual by comparing imagescaptured by the wearable device with the images and the associated datarecords stored in the database of server 250 as described above. Whenthe likelihood for the image of the person obtained from captured imagesmatching the data record from the database is above a predeterminedthreshold value, the hearing aid system may establish that theindividual captured in the images is a recognized individual.

In some cases when the likelihood is insufficiently high (e.g., belowthe predetermined threshold value) the hearing aid system may beconfigured to suggest various possible names for the person displayed inthe captured one or more images. The hearing aid system may then allowuser 100 to select a name that user 100 believes matches best the persondisplayed in the captured images. For cases when the hearing aid systemincludes a display (e.g., a mobile phone a tablet, device 120 withdisplay 260, as shown in FIG. 2, or the like), the hearing aid systemmay cause an image of the at least one individual to be shown on thedisplay. In some embodiments, the hearing aid system may present user100 with one or more images of individuals associated with one or moresuggested possible names for the person displayed in the captured one ormore images. For example, the hearing aid system may show the one ormore images of individuals on display 260 of device 120. Additionally,the hearing aid system may inform user 100 about other informationrelated to individuals associated with one or more suggested possiblenames (e.g., estimated/expected locations of the individuals,occupations of the individuals, etc.) in order to facilitate user 100 inselecting the name of an individual that user 100 believes matches bestthe person displayed in the captured images.

In some cases, the display may be included with a housing common to thewearable camera and the at least one microphone. In some cases, thewearable camera and the at least one microphone may be included in thecommon housing, and the display may be located elsewhere. In someembodiments, the common housing may further include a processor. In somecases, the hearing aid system may include various elements and devicesthat may not be included in the common housing. For example, the hearingaid system may include a second processor that is not included in thecommon housing. In some embodiments, the at least one processor isconfigured to receive the captured images via a wireless link between atransmitter in the common housing and receiver in the second housing.For example, the second housing may be associated with a paired (e.g.,connected wirelessly or wired using any suitable approach) mobiledevice. As previously described, the display may be part of the secondhousing (e.g., a mobile device such as smartphone, tablet, laptop, andthe like) paired with the hearing aid system.

In an example embodiment, the image of the at least one individual thatmay be shown on display 260 may be retrieved from a database stored inmemory, (e.g., the database of server 250) that associates therecognized individuals with corresponding images or features extractedfrom images, as described above. In some cases, the displayed image ofthe at least one individual may be extracted (e.g., derived) from the atleast one image.

For cases when at least one individual is determined to be a recognizedindividual, the hearing aid system may be configured to inform user 100that the individual has been recognized. For example, for cases when thehearing aid system includes a display, the hearing aid system may causean image of the at least one individual to be shown on display (e.g.,display 260 of device 120).

In some cases, the hearing aid system may be configured to displayinformation obtained from the data record associated with the recognizedindividual such as the individual name, address, relationship to user100, and the like. Additionally, or alternatively, the hearing aidsystem may be configured to notify user 100 that the individual has beenrecognized using audio signals delivered to user 100 using any suitablemeans (e.g., using one or more earpiece devices, a speaker, and thelike). For example, the hearing aid system may inform user 100 via oneor more earpiece devices the information obtained from the data recordassociated with the recognized individual, such as the individual name,address, relationship to user 100, and the like. Additionally, oralternatively, the hearing aid system may inform user 100 that theindividual has been recognized using any other suitable approaches(e.g., via a text message, a tactile signal, and the like).

The hearing aid system may selectively condition at least one audiosignal that is received from the at least one microphone and determinedto be associated with the recognized individual. Selective conditioningof an audio signal may involve filtering a selected audio signal fromthe audio signal. In some cases, selective conditioning may includeattenuating the audio signal. Alternatively, selective conditioning mayinclude amplification of the audio signal. In an example embodiment, theselected audio signal may correspond to audio related to theconversation of user 100 with another person. In some cases, the audiosignal may include environmental noises (e.g., various background soundssuch as music, sounds/noises from people not participating inconversation with user 100, and the like), and selected audio signal mayinclude speech of the person participating in the conversation with user100 (referred to as a speaker). In some embodiments, the selectiveconditioning may include changing a tone associated with the at leastone audio signal or changing a rate of speech associated with the atleast one audio signal.

Separating the voice of the speaker from the background sounds may beperformed using any suitable approach, for example, using a multiplicityof wearable microphones mounted at different positions on user 100. Insome cases, at least one microphone may be a directional microphone or amicrophone array. For example, one microphone may capture backgroundnoise, while another microphone may capture an audio signal comprisingthe background noise as well as the voice of a particular person. Thevoice may then be obtained by subtracting the background noise from thecombined audio. In some cases, some of the microphones capable oftransmitting audio to the hearing aid system may be wearable by theperson who is speaking (e.g., a speaker). For example, user 100 may handthe person who is speaking a removable microphone. In some situations,there may be two or more persons engaged in a conversation with user100, with or without background noise. For example, FIG. 33 shows user100, wearing an image capturing device 3322 and an audio capturingdevice 3323, interacting with a speaker 3302 and a speaker 3303. In suchsituations, knowing the identity of at least one of the speakers, or thenumber of speakers, may be helpful in separating the voices.

The number of speakers may be obtained using, for example, a speakerestimation algorithm. The algorithm may receive image data (e.g., animage of speaker 3302 and an image of speaker 3303 as captured byapparatus 110), and based on the received images, output whether theconversation includes multiple speakers. Speaker 3302 and speaker 3303may be identified and recognized by the hearing aid system by finding apair of people facing each other. Multiple images may be captured by thehearing aid system to ensure that the pair of people continue to faceeach other over a period of time. In some embodiments, the hearing aidsystem may identify speakers 3302 and 3303 are engaged in conversationwith user 100 based on the orientation of their faces, gestures of thespeakers (e.g., nodding by one of the individuals when the second personis speaking), the timing of the gestures and sounds, etc. In someembodiments, at least one of the speakers (e.g., speaker 3302) may beidentified by his or her voiceprint. In some embodiments, user 100 mayassist the hearing aid in determining the number of speakers by using apositioning of user 100 head and/or head gestures. The speakerestimation algorithm may output whether the conversation includes nospeech (e.g., only background noise is present), a single speaker, ormultiple speakers.

The head positioning and/or head gestures may be used to determine thenumber of speakers, and also to determine which audio signal isassociated with which speaker. In various embodiments, head positioningfor user 100 may include orienting a face of user 100 towards a speakerthat is talking (e.g., speaker 3303, as shown in FIG. 33), andmaintaining such position for at least a predetermined duration of time(e.g., for a second, for a few seconds, or for a duration of a speech ofspeaker 3303).

In some embodiments, the hearing aid system may be configured to use thecorrelation between head positions of user 100 audio signals receivedfrom speaker 3302 and 3303 to establish the number of speakers for theconversation. Additionally, or alternatively, head gestures such asnodding, head shaking, specific head movements, facial movements, etc.,may also be used to indicate to the hearing aid system the number ofspeakers in the conversation and which audio signal is associated withwhich speaker.

In some embodiments, attributes of an audio signal (e.g., signal fromspeaker 3302 and speaker 3303) may be used alone or in combination withimage data as well as head positioning data and head gestures todetermine the number of speakers in the conversation, and which audiosignal is associated with which speaker. For example, if an audio signalincludes a first audio signal having a first distinct tone, cadence,loudness, etc., and a second audio signal includes a second distincttone, cadence, loudness, etc., the hearing aid system may determine thatthere are two speakers in the conversation. Furthermore, the hearing aidsystem may differentiate between the first and the second audio signalwhen these signals are not overlapping (e.g., when speaker 3302 and 3303are not talking at exactly the same time, which is a typical situationduring a conversation).

In some embodiments, speech content or speech cadence of one of thespeakers (e.g., the speaker 3302) may be analyzed by the hearing aidsystem to differentiate between voices of speaker 3302 and 3303. Forexample, the hearing aid system may determine, based on the content ofthe speech or speech cadence, that speaker 3302 may be awaiting aresponse from speaker 3303. For example, such a situation may arise whenspeaker 3302 asks the speaker 3303 a question or requests informationfrom speaker 3303. In some embodiments, some of the keywords may bedetected by the hearing aid system that may indicate that speaker 3302is awaiting a response from speaker 3303 (e.g., keywords may include“tell us about,” “what do you think about,” etc.). In some cases, thecontent or cadence of the speech of speaker 3302 may indicate thatspeaker 3302 is planning to continue speaking. For example, speaker 3302may use phrases such as “I disagree with you because,” “the listincludes five items, the first item being,” etc.

In various embodiments, the hearing aid system may be configured torecord or transcribe the conversation between multiple speakers. Thetranscription process may be assisted by captured images by the hearingaid system. For example, the hearing aid system may identify andrecognize speaker 3302 and/or speaker 3303. Speaker 3302 may be facingspeaker 3303 (not shown in FIG. 33), and, based on the images capturedby image capturing device 3322 of the hearing aid system, the hearingaid system may determine that speaker 3302 is addressing speaker 3303.The hearing aid system may be configured to transcribe the conversationbetween speaker 3302 and speaker 3303 and to identify the first speechas belonging to speaker 3302 and the second speech as to belonging tospeaker 3303.

In various embodiments, a voiceprint of a speaker may be obtained usingan audio signal associated with a speech of the speaker and stored inthe database of server 250 for further reference. The stored voice datamay include one or more voiceprints that may be obtained from one ormore speeches of the speaker. In an example embodiment, at least oneaudio signal may be determined to be associated with the recognizedindividual based on one or more predetermined voiceprint characteristicsassociated with the recognized individual detected in the at least oneaudio signal. The predetermined voiceprint may be stored in associationwith a person and one or more images or visual characteristics thereof,and optionally updated over time, enhanced, or the like. When thespeaker is recognized in one or more images, one or more voiceprints maybe retrieved and used for separating the specific voice from a mixtureof voices. In an example embodiment, the voiceprint may be stored in thedatabase of server 250 and may be associated with the data recordcorresponding to the speaker. Additionally, the voiceprint may furtherbe associated with one or more images of the speaker related to the datarecord.

Alternatively, for example, if a speaker is not identified, thespeaker's voiceprint may be extracted from an earlier part of theconversation when only that speaker was engaged in the conversation. Theextraction of the voiceprint may be performed on segments of the audiofor which the number of speaker algorithm indicates a single speaker.The extracted voiceprint may then be used later in the conversation forseparating the speaker's voice from other voices. The separated voicecan be used for any purpose, such as transmission over the phone,transmission to a microphone, transmission to a hearing aid, or thelike.

In some cases, the hearing aid system may be configured to obtain afirst audio sample from the first speaker (e.g., speaker 3302) separatedfrom a second audio sample from the second speaker (e.g., speaker 3303).The hearing aid system may use the first audio sample to determine afirst voiceprint for speaker 3302 and the second audio sample todetermine a second voiceprint for speaker 3303. As described above, aspeaker communicating with user 100 may be identified using imagescaptured by an apparatus such as apparatus 110. An individual may beidentified as the speaker if the speaker is located in the center of theuser's field of view as captured by a wearable camera of the hearing aidsystem. In other embodiments, the speaker may be identified as a speakerto which the user's chin, as recognized in one or more images, isdirected.

The voiceprint extraction may be facilitated by user 100 head positionand/or head gestures. For example, at the beginning of a conversation,user 100 may orient his/her face towards a speaker that is talking touser 100 as shown, for example, in FIG. 33, by looking at speaker 3303.Similarly, when speaker 3302 is talking, user 100 may look at speaker3302 to indicate to the hearing aid system that an audio signal receivedby the hearing aid system is primarily due to the speech of speaker3302. In an example embodiment, at the beginning of the conversation,the hearing aid system may not be configured to separate the specificvoice from a mixture of voices prior to obtaining sufficient data (e.g.,voiceprint related data) to adequately separate voices. However, oncethe hearing aid system receives sufficient information to adequatelyseparate voices, the hearing system may selectively condition (e.g.,abruptly or gradually) the audio signal related to the conversation ofuser 100, by separating a voice of a speaker engaged in a conversationwith user 100.

In various embodiments, a speaker's voiceprint and a high-qualityvoiceprint, in particular, may provide for fast and efficient speakerseparation. A high-quality voiceprint for a speaker may be collected,for example, when the speaker speaks alone, preferably in a quietenvironment. Having a voiceprint of one or more speakers, a processor ofthe hearing aid system to separate an ongoing voice signal almost inreal time, e.g., with a minimal delay, using a sliding time window. Thedelay may be, for example, 10 milliseconds, 20 milliseconds, 30milliseconds, 50 milliseconds, 100 milliseconds, or the like. Differenttime windows may be selected, depending on the quality of thevoiceprint, on the quality of the captured audio, the difference incharacteristics between the speaker and other speaker(s), the availableprocessing resources, the required separation quality, or the like.

A voiceprint extraction may be performed by extracting spectralfeatures, also referred to as spectral attributes, spectral envelope, orspectrogram from clean audio of a single speaker. The clean audio mayinclude a short sample (e.g., one second long, two seconds long, and thelike) of the voice of a single speaker isolated from any other soundssuch as background noises or other voices. The clean audio may be inputinto a computer-based model such as a pre-trained neural network, whichoutputs a signature of the speaker's voice based on the extractedfeatures. Such signature of the speaker's voice may include the sameinformation as a voiceprint for the speaker. Additionally, oralternatively, the signature of the speaker's voice may include audioinformation that may be used to obtain the voiceprint for the speaker.In some cases, the signature of the speaker's voice may include audioinformation that can be used to obtain at least some of the data neededto determine the voiceprint for the speaker.

The output signature may be a vector of numbers. For example, for eachaudio sample submitted to a computer-based model (e.g., a trained neuralnetwork), the computer-based model may output a set of numbers forming avector. Any suitable computer-based model may be used to process theaudio data captured by one or more microphones of the hearing aid systemto return an output signature. In an example embodiment, thecomputer-based model may detect and output various statisticalcharacteristics of the captured audio such as average loudness oraverage pitch of the audio, spectral frequencies of the audio, variationin the loudness, or the pitch of the audio, rhythm pattern of the audio,and the like. Such parameters may be used to form an output signaturecomprising a set of numbers forming a vector.

The output signature may be a first vector representing the speaker'svoice, such that the distance between the first vector and anothervector (i.e., another output signature) extracted from the voice of thesame speaker is typically smaller than the distance between the outputsignature of the speaker's voice and the output signature extracted froma voice of another speaker. In some embodiments, output signature of thespeaker's voice may be a voiceprint for the speaker and may include asound spectrogram that may be a graph that shows a sound's frequency onthe vertical axis and time on the horizontal axis. Different speechsounds may create different shapes within the graph. The voiceprint maybe represented visually and may include colors or shades of grey torepresent the acoustical qualities of a sound of the speaker's voice.

FIG. 34A shows a flowchart of a process 3400 for separating voices in anaudio signal. At step 3451, an audio signal 3401 may be received by ahearing aid system. The hearing aid system may include a computer-basedmodel 3403 for separating an audio signal that corresponds to a voice ofa speaker from the background sounds using any suitable approachesdescribed above. In some cases, the hearing aid system may record a“room tone” prior to the beginning of the conversation, where room tonemay refer to the natural noise of the environment of user 100. The audiosignature corresponding to room tone may be used to filter out thebackground noise from an audio signal containing conversational sounds.

Model 3403 may output a voice audio signal 3404 at step 3452. At step3453, signal 3404 may be received by a voice model 3410, and at step3454 the voice model may output voiceprint 3411 for the speaker's voice.Model 3410 may use any suitable approach describing above for obtainingvoiceprint 3411 from the speaker's voice, such as extracting aspectrogram from the speaker's voice, extracting statistical audiocharacteristics, and the like. At step 3455 of process 3400, acomputer-based model 3430 may receive voiceprint 3411 and an audiosignal 3421 that may include background sounds, and/or one or morevoices of one or more individuals.

In various embodiments, computer-based model 3430 may be a neuralnetwork. At step 3456, model 3430 may receive the noisy audio signal3421 and the speaker's signature or voiceprint 3411, and, at step 3457Aor/and 3457B, output audio signal related to a voice 3431A and/or avoice 3431B. It should be noted, that filtered (i.e., separated) voice3431A and/or 3431B may be used to prepare an additional voiceprint(or/and output signature) for the speaker that can be used bycomputer-based model 3430. In some embodiments, more than one voiceprint(e.g., voiceprint 3411) may be used as an input for model 3430 at step3455. In some embodiments, multiple voiceprints may correspond to thesame individual, and in other embodiments, some of the voiceprints maycorrespond to a first person (e.g., speaker 3303, as shown in FIG. 33)and other voiceprints may correspond to a second person (e.g., speaker3302, as shown in FIG. 33).

FIG. 34B shows an illustrative process 3470 for separating voice signalfrom an audio signal using a video signal. At step 3461, model 3445 mayreceive data 3443 related to a conversation of user 100 with speakers3302 and 3303. Data 3443 may include a video signal 3441 and an audiosignal 3421. Video signal 3441 may indicate whether a speaker (e.g.,speaker 3303) is talking or silent. For example, video signal 3441 mayshow the lips movement of speaker 3303. Audio signal 3421 may include abackground sound as well as voices of speaker 3302 and 3303 that may ormay not overlap. For example, the voice of speaker 3302 may overlapbriefly with the voice of speaker 3303. Model 3445 may identify, andseparate voices for speaker 3302 and 3303 by synchronizing the lipsmovement of speaker 3303 (or speaker 3302) with words/sounds identifiedin audio signal 3421. In some embodiments, the hearing aid system may beconfigured to collect video and audio data for both of the speakers. Forexample, the hearing aid system may be configured to detect lipmovements of speaker 3302 and speaker 3303 during the conversation. Atstep 3457A or/and 3457B, model 3445 may output audio signal related to avoice 3431 and/or a voice 3432.

In various embodiments, the selectively conditioned audio signal (e.g.,voice 3431A or voice 3431B) processed by the hearing aid system asdescribed in process 3400 or process 3470 may be transmitted to aninterface device (e.g., an earpiece, headphones, a speaker, a display, avibrational or tactile device, etc.) for delivering the audio signal touser 100. In various embodiments, the interface device may be part ofthe hearing aid system. In an example embodiment, the interface devicemay transmit to user 100 an audio signal (e.g., a signal transmitted touser 100 via an earpiece), a visual signal (e.g., a text written on ascreen or a video of a person communicating with user 100 via a silentlanguage), a vibration signal, a tactile signal (e.g., tactile lettersused by visually impaired people for reading), an electric signal, andthe like. In an example embodiment, the interface device may include ahearing interface device configured to provide sound to an ear of theuser. In an example embodiment, the hearing interface device may includea speaker associated with an earpiece or a bone conduction microphone.

In some embodiments, the hearing interface device of the hearing aidsystem may transmit to user 100 an audio signal corresponding to aspeech of a speaker as it is extracted from an audio signal captured byone or more microphones of the hearing aid system. Additionally, oralternatively, the hearing aid system may be configured to modify one ormore parameters of the audio signal corresponding to the speech of thespeaker. For example, the hearing aid system may modify the pitch,loudness, cadence, etc., of the audio signal prior to providing thesignal to user 100 via the hearing interface device. In someembodiments, the hearing aid system may be configured to transcribe thespeech of the speaker, modify the transcribed speech, and read thetranscribed speech using a text-to-speech natural voice artificialintelligence reader. In some embodiments, when multiple voices aredetected (e.g., when voices 3431A and 3431B overlap), the hearing aidsystem may be configured to time shift one voice relative to another toreduce the overlap. Alternatively, the hearing aid system may beconfigured to modify one or more characteristics of one of the voices(e.g., voice 3431A) to further differentiate it from voice 3431B. Forexample, the tone, cadence, etc. of voice 3431A may be modified todifferentiate voice 3431A from voice 3431B.

In some embodiments, when multiple microphones are present (e.g., twomicrophones are present), a delay of the audio signal measured betweenthe two microphones may be used to determine the directionalcharacteristics of the audio signal related to the speaker's voice. Inan example embodiment, user 100 may have a left microphone positionednext to a left ear and a right microphone positioned next to a rightear. A left speaker, engaged in conversation with user 100, may bepositioned slightly to the left of user 100, and audio signals from theleft speaker may arrive first in the left microphone, and second, to theright microphone, resulting in a phase shift between audio signalsreceived by the two microphones. The phase shift may be used todistinguish from other signals that may not have a well-defined phaseshift, or which may have a different phase shift. For example, ifanother speaker is present (e.g., a right speaker positioned slightly tothe right of user 100), audio signals from the right speaker may have adifferent phase shift compared to the phase shift for the left speaker.For example, the audio signals from the right speaker may arrive firstto the right microphone of user 100, and after that, to the leftmicrophone of user 100, resulting in a phase shift that has an oppositesign from the phase shift for the left speaker. In various embodiments,user 100 may be able to move his/her head to further differentiatebetween left and right speaker using the left and the right microphone.

In some cases, an audio signal from different speakers cannot beseparated by the hearing aid system due to, for example, an unavailablevoiceprint for the one or more speakers, a low-quality voiceprint forthe one or more speaker, two or more voices of the different speakersbeing similar to each other, or the like. In such cases, when it isdetermined that there are two or more voices present in a conversation,the output signal may be silenced. This feature may help user 100 adaptin a noisy environment. For example, such a feature may prevent user 100from hearing loud and unpleasant noise while not being able tounderstand what is being said. Thus, silencing the output signal, asdescribed above, may not reduce user 100 understanding of theconversation, but may reduce the environmental noise and thus improveuser 100 with a comfort level during the conversation. In variousembodiments, the silencing of the output signal can be partial (alsoreferred to as suppression of the output signal).

FIG. 35A shows an illustrative process 3500 for transmitting a voiceseparated from an audio signal to a device, such as earpiece of thehearing aid system. At step 3504, one or more images captured by theapparatus described above may be received by a processor of a hearingaid system or by a processor of server 250. In some embodiments, theimages may be captured substantially in line with the user's line ofsight, such that a speaker communicating with user 100 is at or near thecenter of the image. At step 3508, the speaker may be identified in thecaptured images using any suitable approaches as described above. Atstep 3512, the speaker may be recognized. The speaker identification mayrelate to locating a person within an image, while recognition of thespeaker may relate to recognizing the identified person as being aspecific known person. The person may be recognized if previouslycaptured by the device, and his or her name or another detail wereprovided by a user or in any other manner. The recognition of thespeaker may be done using any suitable approaches described above. Atstep 3514, an image of the recognized individual may be shown on adisplay associated with the hearing aid system. For example, an image ofthe recognized individual may be retrieved from the database anddisplayed on a mobile device paired with the hearing aid system.

At step 3516, an audio signal may be received by a microphone of thehearing aid system (e.g., an audio signal of user 100 communicating withanother individual). The audio signal may be further analyzed by aprocessor of the hearing aid system. In an example embodiment, theprocessor may be configured to determine if the captured audio signalcorresponds to a recognized individual. For example, the processor maybe configured to retrieve the recognized individual voiceprint, from astorage device, based to the person's identity. For example, thevoiceprint may be retrieved from the database of server 250.Additionally, or alternatively, the voiceprint may be obtained byanalyzing a speech of the speaker during a conversation that does notcontain audio signals from other speakers, and/or does not contain ahigh volume of environmental noises. For example, a conversation may befirst conducted in a quiet environment (e.g., a car prior to arriving atan event) followed by a conversation at a restaurant (e.g., a noisyenvironment).

At step 3520, the received audio signal may be selectively conditionedby the processor. Additionally, or alternatively, the received audiosignal may be transmitted to server 250 and selectively conditioned byone of the processors of server 250. In an example embodiment,selectively conditioning may include separating voices of one or morespeakers from the audio signal. Separation may be performed using any ofthe approaches described above. In some embodiments, if the speaker isrecognized at step 3512, and the voiceprint of the speaker is obtainedat step 3602, the received audio may be separated by extracting only thevoice of the particular speaker rather than all voices participating inthe conversation or background voices (e.g., voices that do not directlyengage user 100). Selective conditioning may use any of the suitableapproaches discussed above.

At step 3524, the speaker's voice may be provided to the hearing aidsystem of user 100 to help user 100 focusing on the conversation withthe speaker while reducing disturbances from the environmental noisesand/or from other background voices. As previously described, the audiosignal related to the voice of the speaker may be altered by the hearingaid system (e.g., the voice may be amplified, and/or otherwise modified,for example, by changing the tone of the voice, using noise cancellationor other approaches).

In various embodiments, the audio processing may be combined with animage processing technique, for example by identifying, and in some caserecognizing a speaking person, synchronizing received audio with amotion of lips of a speaker in communication with user 100 and/or lipreading based on the motion of the speaker's lips (e.g., the sound “ba”may be uttered by a person who opened his/her mouth). In anotherexample, if no person in an environment of user 100 is speaking, thenbackground noise may be detected and canceled.

In some embodiments, the audio signal received by user 100 may arrive touser 100 via different channels. For example, user 100 may participatein a phone audio/video conversation with a speaker, and backgroundnoises may be due to an environment of user 100. In such cases, thebackground noises may be suppressed.

In various embodiments, the hearing aid system may be operated by abattery. In order to prolong the functioning of the hearing aid system,various approaches may be used to reduce the power consumption of thehearing aid system. For example, the hearing aid system may optimize arate of capturing video frames, reduce the resolution of capturedimages, optimize compression of the captured images and/or optimizecompression/quality of a captured audio signal. Other steps for reducingpower consumption by the hearing aid system may include optimizing theprocess of data transfer from the hearing aid system to server 250. Forexample, the hearing aid system may be configured to transfer data toserver 250 periodically, with time intervals between data transfersincreasing when a reduction in power consumption by the hearing aidsystem occurs.

As described, in various embodiments, the voice audio signalcorresponding to a speech of a speaker may be manipulated prior totransmitting it to user 100. For example, if a rate of the speechexceeds a predetermined value, the speech may be slowed and transmittedto the hearing aid system at a lower rate. The lower rate may becompensated during breathing or other pauses, so as not to accumulatedelay. In further embodiments, slang or inappropriate words may bereplaced. Such features may be useful, for example, in helping olderpeople communicate with younger ones, for example, their grandchildren.In some embodiments, slower speech may be accelerated, which may helpprevent boredom or allow a user to more rapidly listen to audio.

In some embodiments, the database may be configured to establish atimeline for various encounters with various speakers and track theencounters with different individuals chronologically. In some cases,based on input received from user 100, the one or more processors of thehearing aid system may be configured to forego transmitting to thedatabase information related to encounters with one or more individualsidentified in the plurality of images, thus, preventing the storage ofsuch information. For example, the information related to the encountermay be ignored (i.e., not stored in the database of server 250) if user100 believes the encounter is not important, or if she/he prefers theinformation not to be available for access/inspection later by a thirdparty and/or by user 100. In some cases, to prevent access to theinformation stored in the database, the information may be passwordprotected.

In some instances, the one or more processors of the hearing aid systemmay be configured to forego transmitting to the database informationrelated to encounters with one or more individuals determined to beassociated with a predetermined group of individuals. For example, agroup of individuals may be identified by identifying members of thegroup or by identifying attributes of individuals in the group (e.g.,all the individuals that wear a uniform). The attributes of theindividuals may be identified by user 100 and entered using a userinterface for the hearing aid system. In some cases, the attributes ofthe individuals may be inferred from the images captured by the hearingaid system. In example embodiments, the one or more predetermined groupsof individuals may include office workers, service personnel, or variousindividuals with whom user 100 does not participate in vocalinteractions. In some embodiments, the predetermined group may includeindividuals who do not participate in a conversation with user 100, andin some embodiments, the predetermined group may include individuals whodo not participate in a conversation with individuals that participatein a conversation with user 100. The information related to theencounter with one or more individuals determined to be associated witha predetermined group of individuals may be ignored (i.e., not stored inthe database of server 250) if user 100 believes the encounter is notimportant, or if she/he prefers the information not to be available foraccess/inspection later by a third party and/or by user 100. In somecases, to prevent access to the information stored in the database, theinformation may be password protected.

FIG. 35B is an illustrative process 3550 for recording an encounter in atimeline. At step 3551 of process 3550, the hearing aid system may beconfigured to capture the encounter with an individual for user 100using any suitable approaches described above. For example, the hearingaid system may be configured to capture the encounter by directing acamera and a microphone of the hearing aid system to capture theimage/video data and audio data related to the encounter. At step 3553,the hearing aid system may obtain input from user 100 whether theencounter should be ignored. For example, the hearing aid system mayobtain an audio signal from user 100 to ignore the encounter.Additionally, or alternatively, the hearing aid system may obtain user100 input regarding ignoring the encounter via a touch screen (e.g., atouch screen of a mobile device paired with the hearing aid system). Ifthe encounter is determined not to be ignored (step 3553, No), theencounter may be recorded in a timeline at step 3555. Recordation of theencounter may allow user 100 to retrieve information associated with theencounter by specifying some identifying characteristics of theencounter, such as date and time of the encounter, the nature of theencounter, the speaker identified and recognized in the encounter, thetopic of conversation, etc. If the encounter is determined to be ignored(step 3553, Yes), the encounter may not be recorded, and process 3550may be terminated. In some embodiments, if the encounter is determinedto be ignored (step 3553, Yes), the encounter may be recorded for apredefined period of time before it is deleted from the timeline. Suchtemporal recordation of the encounter may allow user 100 to changehis/her mind regarding ignoring the encounter.

Selectively Conditioning Audio Signals Including Overlapping Voices

In some embodiments, a hearing aid system may include at least oneprocessor programmed to receive audio signals from the at least onemicrophone. The processor may be configured to detect, based on analysisof the audio signals, a first audio signal associated with a first timeperiod, wherein the first audio signal is representative of a voice of asingle individual. In addition, the processor may be configured todetect, based on analysis of the audio signals, a second audio signalassociated with a second time period, wherein the second time period isdifferent from the first time period, and wherein the second audiosignal is representative of overlapping voices of two or moreindividuals.

The audio signal corresponding to the overlapping voices may include atleast two overlapping voices, and in some cases, may include more thantwo voices. In some cases, some of the overlapping voices may be inclose proximity to user 100, and have a high amplitude, while otheroverlapping voices may be further away from user 100 and have a loweramplitude. In some cases a voice with a high amplitude may overlap avoice with a lower amplitude. In various embodiments, a first and asecond voice overlap when a sound associated with the first voice isemitted during a first time interval, a sound associated with the secondvoice is emitted during a second time, and the first and the second timeintervals overlap. In some cases, the first and the second time windowsmay overlap partially. For example, a portion of the first time windowmay overlap with a portion of the second time window. It should be notedthat, the duration of the first time window may be shorter or longerthan a duration of the second time window.

When the audio signal contains more than two voices, more than twovoices may overlap. For example, when audio signal includes a first, asecond, and a third voice, the first and the second voice may overlap,the second and the third voice may overlap, the third and the firstvoice may overlap, and, in some cases, all three voices may overlap.

In an example embodiment, the processor may selectively condition thefirst audio signal and the second audio signal, wherein the selectiveconditioning of the first audio signal may be different in at least onerespect relative the selective conditioning of the second audio signal.For example, selectively conditioning the first audio signal may includeremoving the background sound and separating the voice of the individualdetected in the first audio signal. Additionally, or alternatively,selective conditioning the first audio signal may include amplificationof the signal, changing a tone of the signal, or changing a rate ofspeech associated with the signal. In an example embodiment, selectivelyconditioning the second audio signal may include amplification of thesecond audio signal. In some cases, when both the first and the secondaudio signals are amplified, the amplification level associated with thesecond audio signal may be less than the amplification level associatedwith the first audio signal. In some embodiments, the selectiveconditioning of the second audio signal includes attenuation of thesignal. For example, the signal may be completely or partiallyattenuated. In some cases, some of the frequencies of the second audiosignal may be attenuated. Additionally, or alternatively, some of theportions of the second audio signal may be attenuated while otherportions may be unchanged or amplified. In some cases, the amplitude ofthe second audio signal may be attenuated in a time-dependent way, andin some cases, an amplitude of a set of frequencies of the second audiosignal may be attenuated in a time-dependent way. In some cases, theselective conditioning of the second audio signal includes foregoingtransmission of the second audio signal to the hearing interface deviceconfigured to provide sound to an ear of the user. For example, thesecond audio signal may not be transmitted to the hearing interfacedevice when voices in the second signal are not clearly discernible andmay cause confusion to user 100.

In various embodiments, at least one processor of the hearing aid systemmay be programmed to analyze the plurality of images captured by awearable camera (e.g., image capturing device 3322) and identify in atleast one of the plurality of images representation of a singleindividual associated with the first audio signal. For example, theprocessor may be programmed to capture a video data related to facialexpressions of the individual and analyze video frames by evaluating thecorrelation between facial expressions and sounds detected in the firstaudio signal. In some cases, the analysis may identify correlationsbetween particular facial expressions and particular sounds or soundfluctuations. For example, a facial expression related to a particularlip movement may be associated with a sound or word that may have beensaid during a conversation captured in the first audio signal. In someembodiments, the analysis of the plurality of images is performed by acomputer-based model such as a trained neural network. For example, thetrained neural network may be trained to receive an image and/or videodata related to facial expressions of an individual and predict a soundassociated with the received image and/or video data. For example, thetrained neural network may be trained to receive an image and/or videodata related to facial expressions of an individual and a sound, andoutput whether the facial expressions correspond to the sound. In someembodiments, other factors, such as gestures of the individual, theposition of the individual, the orientation of the individual's face,etc., may be identified in the one or more images captured by a wearablecamera. These factors may be used alone or in combination with facialexpressions of the individual to determine if the individual isassociated with the first audio signal.

In some embodiments, at least one audio signal (e.g., the first audiosignal) is determined to be associated with the recognized individualbased on lip movements of the user, as detected based on analysis of theplurality of images. For example, the first audio signal may bedetermined to be associated with the recognized individual based on adetermination of whether the detected lip movements are consistent witha voice signal associated with the at least one audio signal.

In various embodiments, at least one processor of the hearing aid systemis configured to transmit the conditioned first audio signal to ahearing interface device configured to provide sound to an ear of user100 using any suitable approaches discussed above. In some embodiments,the processor may also be configured to transmit conditioned secondaudio signal to the hearing interface device configured to provide soundto an ear of user 100 using any suitable approaches discussed above.

FIG. 36A shows an illustrative process 3600 describing exemplary stepsfor transmitting a voice audio data to user 100 from a speaker incommunication with user 100. At step 3602 the hearing aid system mayreceive captured audio. Step 3602 of process 3600 may be similar to step3516 of process 3500. At step 3603, the hearing aid system may determineif the received audio includes a speech by a single speaker. Suchdetermination may be done using any suitable approaches described above(e.g., using multiple microphones that may determine a phase shiftcorresponding to an audio signal for the speech of the speaker, using acomputer-based model that can take as an input a voiceprint of thespeaker, an audio analysis algorithm that can evaluate a number ofspeakers, and the like). If the hearing aid system determines that thereceived audio includes a speech from a single speaker (step 3603, Yes),at step 3605 a voiceprint of the speaker may be extracted from theaudio, and optionally used for enhancing a previously availablevoiceprint. In some embodiments, the voiceprint may be associated withthe speaker that is recognized by the hearing aid system, and in someembodiments, the voiceprint may be extracted for the speaker when thespeaker is not recognized. In some embodiments, if a voiceprint existsfor the speaker, then a new voiceprint may not be extracted from thereceived audio data. At step 3607, the speaker's voice may betransmitted to the hearing interface device of the hearing aid systemand delivered to user 100 as an audio signal as described above. In someembodiments, the voice may be transmitted to user 100 via a visualinterface, tactile interface, and the like.

If at step 3603 the hearing aid system determines that the receivedaudio signal includes voices of multiple speakers (step 3603, No),process 3600 may be configured to follow step 3604 and determine if avoiceprint for at least one speaker is available. In variousembodiments, if one or more speaker is recognized by the hearing aidsystem, the hearing aid system may access data associated with the oneor more speaker and determine if the related voiceprint is available.For example, the hearing aid system may access the speaker data recordfrom the database of server 250 to determine if the voiceprint isavailable. Step 3604 may use a trained model to determine whether or notan audio signal comprises speech associated with a particularvoiceprint, or provide a probability that the audio signal comprisesspeech associated with the particular voiceprint.

In some embodiments, once a speaker in communication with user 100 isrecognized and her/his voice audio data is separated from an audiosignal for the conversation and transmitted to user 100, the voice audiodata may be transmitted as long as the speaker is continuously speaking,even if/when other voices, whether recognized or not are captured. Suchan approach may be used to allow user 100 to continuously listen to thespeaker. Short breaks in speaking, for example, breathing breaks orpauses while searching for a word may still be considered as acontinuous speech. In some embodiments, pauses up to a predeterminedlength may be considered as part of continuous speech, while longerperiods may be considered as the end of a speech by the speaker, suchthat other speakers can be detected or amplified.

If a voiceprint is available, (step 3604, Yes) the speaker's voice maybe separated from the audio data and transmitted to user 100 at step3606. If no voiceprint is available, and/or if the separation of thespeaker's voice from the audio data is not successful, (step 3604, No)the hearing aid system may silence the output at step 3601. The outputmay be silenced using any of the approaches described above. In someembodiments, completely silencing the rest of the voices may create anuneasy and out of context feel, for example, when speaking to a personin a restaurant and seeing the waiter approaching and talking but nothearing anything. Therefore, providing a low but positive amplificationfor the other sound, for example, 10%, 20%, or any other suitable degreeof the volume may feel more natural for user 100. Similarly, if no voiceis recognized by the hearing aid system, instead of silencingeverything, the loudness of the environmental noises can be reduced to apredetermined level. In such circumstances, the audio related to theenvironmental sounds may be transmitted at a low volume (e.g., 10% ofthe original volume) for a more natural feeling, enabling user 100, forexample, to hear some background noise at a restaurant. The loudnesslevel selected by the hearing aid system may be set by a user orpredetermined, depending on an environmental situation, location of user100, time of the day, and the like.

The hearing aid system may be configured to process environmental soundsand determine if some of the sounds may or may not be silenced orsuppressed. For example, sounds that may be important and related touser 100 may not be filtered, suppressed, or silenced. For example,emergency sounds, such as a fire alarm, a siren, sounds of screams,sounds of crying kids, etc. may not be silenced, modified, orsuppressed.

In some embodiments, some sounds (whether predetermined, recognized ornot) may be amplified and transmitted at a delay. For example, in anairport when there is an announcement about a flight, the device mayrealize that this is an important announcement only after the flightnumber is mentioned. Then it can play the whole announcement even thoughthe sound of the announcement may not relate to the voice of a speakerengaged in conversation with user 100.

In various embodiments, the voice separation and amplification processmay be time-dependent and may depend on a content of audio captured bythe microphones of the hearing aid system as well as environmentalfactors may be determined, for example, by analyzing the images capturedby the hearing aid system. In some embodiments, the hearing aid systemmay collect audio data during a predetermined sliding time window andthen separate voice audio data within such a window. For a short timewindow (e.g., a millisecond, few milliseconds, a second, few seconds,etc.) user 100 may experience only a short delay, between captured audioand transmitted voice data to user 100. In an example embodiment, thetime window may be less than a second, for example, 10 milliseconds, 20milliseconds, 50 milliseconds, 100 milliseconds, or the like.

In some embodiments, the hearing aid system may enable user 100 toselect an audio voice signal from certain individuals to take precedenceover other audio voice signals. In various embodiments, the voiceprintsfor these individuals may be extracted and stored. If one of theindicated speakers is recognized within an image captured by the hearingaid system, or if their voices are recognized within a captured audio,these selected voices may be amplified over voices of other speakers orother sounds, such as sounds from a television or speaker. For example,the voice of a parent, a spouse, a child, a grandchild, agreat-grandchild, or other family members may be identified andamplified for user 100, enabling user 100 to identify these voices overother voices or other environmental sounds.

In some embodiments, the hearing aid system may interact with variousother devices when audio signals related to voices/sounds of recognizedindividuals (including a voice of user 100) are captured by the hearingaid system. For example, the hearing aid system may interact withsmart-home devices that may affect the environment of user 100. Forexample, the hearing aid system may interact with smart-home devices toturn the light on/off, to make special sounds, to turn the televisionon/off, and the like.

Audio sounds not related to voices may be separated from other audiosignals. For example, such audio sounds may include sounds of a dogbarking or howling, the sound of a baby crying or making sounds, soundsof broken glass, sounds of dropped objects, sounds ofcreaking/opening/closing doors, doorbell sounds, and the like. Thesesounds may be amplified relative to other environmental sounds that maybe less important (e.g., the sound of a radio station).

The amplified voices may be provided to user 100 via a hearing aid,while other sounds may be silenced or suppressed. User 100 may configurethe hearing aid system to silence or suppress voices of selectedindividuals (the selected individuals may be selected, for example, byuser 100 via an interface device for the hearing aid system). Forexample, the selected individuals may be recognized in one or moreimages captured by the hearing aid system, or these individuals may berecognized within captured audio due to their voiceprint. As describedabove, in addition to, or instead of amplifying an audio signal relatedto the selected voices, the audio signal of the voices may be enhanced,for example, the tone may be changed, or other modifications may bemade.

In some embodiments, the hearing aid system may create a “hierarchy” ofvoices for various speakers, wherein the hierarchy may or may not betime or situation-dependent. For example, during a lecture, user 100 maywant the lecturer's voice to be amplified, even when the user is lookingat another person. In an example embodiment, when user 100 is in ameeting, user 100 may want to receive a speech from a selected person ora group of people (e.g., from a supervisor of user 100). In someembodiments, user 100 may separate various voices within the capturedaudio data but amplify only one selected voice. In various embodiments,other voices may be recorded or/and transcribed. In some cases, user 100may discard the voices of some individuals that are deemed unimportant.

In various embodiments, the hearing aid system may be apparatus 110 thatmay include a processor and a memory as described above. The hearing aidsystem may include software applications that may contain softwaremodules stored in a memory 3650 of the hearing aid system asschematically shown in FIG. 36B. The software modules may include aspeaker identification module 3654, a speaker separation module 3658, aspeaker and speech matching module 3662, a voiceprint handling module3666, and a user interface 3670.

Modules 3654, 3658, 3662, 3666, and 3670 may contain softwareinstructions for execution by at least one processing device, e.g.,processor 210, included with a wearable apparatus. In some embodiments,any one or more of the modules may facilitate processing one or moreimages from an image sensor, e.g., image sensor 220 and audio sensor togenerate a set of instructions to assist user 100 in improving user 100hearing voice of one or more speakers.

Speaker identification module 3654 can be used in identifying one ormore speakers in an image captured by the apparatus, such that a speakercommunicating with user 100 may be identified. Speaker identificationmodule 3654 can identify the speaker by his/her location in the capturedimages that may display a field of view of user 100. For example, if thespeaker is in the center of the image, the speaker may be in the centerof the field of view. In some embodiment, the direction of user 100 headmay be used to identify a speaker. For example, the hearing aid systemmay be configured to capture an image in a direction normal to a face ofuser 100 (e.g., normal to a chin of user 100) to discern a speaker inthe captured image. In some embodiments, module 3654 may identify thedata record for the speaker, as previously discussed. In an exampleembodiment, module 3654 may associate at least one audio signal with theidentified and recognized individual based on a detected look directionfor the user, determined based on a direction associated with a chin ofthe user detected in the at least one of the plurality of images.

Voiceprint handling module 3666 may be used to generate, store, orretrieve a voiceprint, using, for example, wavelet transform or anyother attributes of the voice of one or more persons. Voiceprinthandling module 3666 may use any suitable algorithm to determine whetheran audio signal comprises speech, as well as to determine whether thereis one speaker or multiple speakers participating in a conversation. Avoiceprint may then be extracted from single speech audio using a neuralnetwork as described above. Information obtained using voiceprinthandling module 3666 may be transmitted to speaker identification module3654, and module 3654 may associate the identified speaker with thevoiceprint.

In various embodiments, speaker separation module 3658 may receive noisyaudio captured by the device and voiceprints of one or more speakers,and separate one or more voices for one or more speakers using any ofthe methods described above. In some embodiments, for example when novoiceprint is available for the speaker, matching the voice with aspecific speaker may be performed in accordance with the capturedimages, for example by matching identified words with the lip movementof the speaker, by matching speaking and silent periods, or the like.

In some embodiments, when the identity of the speaker is determined fromthe captured images, speaker and speech matching module 3662 may be usedto match the identity of the speaker with the audio signal correspondingto the voice of the speaker detected using speaker separation module3658. In some embodiments, when speaker identity is not established, animage of the speaker may be used by module 3662 to correspond with theaudio signal of the voice of the speaker.

In various embodiments, the hearing aid system may include userinterface 3670 to allow user 100 to change performance characteristicsof the hearing aid system. In some embodiments, the user interface 3670may include an interface for receiving a visual, audio, tactile, or anyother suitable signal from user 100. For example, the interface mayinclude a display that may be part of a mobile device (e.g., asmartphone, laptop, tablet, etc.) In an example embodiment, theinterface may include a touch screen, a graphical user interface (GUI)having GUI elements that may be manipulated by user gestures, or byappropriate physical or virtual (i.e., on screen) devices (e.g.,keyboard, mouse, etc.). In some embodiments, interface 3670 may be anaudio interface capable of receiving user 100 audio inputs (e.g., user100 voice inputs) for adjusting one or more parameters of the hearingaid system. For example, user 100 may adjust the loudness of the audiosignal produced by the hearing aid system using audio voice inputs, thepitch of the audio signal produced by the hearing aid system, tempo ofthe audio signal, and the like. In some embodiment, user interface 3670may be configured to assist user 100 in identifying the data record fora speaker in conversation with user 100 and for facilitating separationof the voice of the speaker from the audio data captured by microphonesof the hearing aid system. For example, interface 3670 may prompt user100 to select a name for the speaker from a list of available names, todisplay an image of the speaker, to select an audio stream correspondingto the voice of the speaker, and the like.

FIG. 37A shows an illustrative process 3700 of transmitting aconditioned audio signal to a device (e.g., an earpiece of the hearingaid system). At step 3504, one or more images of a speaker engaged in aconversation with user 100 may be captured by the hearing aid system.Step 3504 of process 3700 may be the same as step 3504 of process 3500.At step 3508, the speaker may be identified as described above. Invarious embodiments, step 3508 of process 3700 may be the same as step3508 of process 3500. At step 3701, the hearing aid system may determineif the identified speaker is recognized using any of the suitableapproaches described above. If the speaker is recognized (step 3701,Yes) the image of the recognized person may be displayed in step 3703.If the speaker is not recognized (step 3701, No) process 3700 may beterminated. In some embodiments, if the speaker is not recognized, theaudio signal associated with the voice of the speaker may be silenced orsuppressed as described above.

At step 3516, a processor of the hearing aid system may receive audiodata related to user 100 conversing with one or more speakers. Invarious embodiments, step 3516 of process 3700 may be the same as step3602 of process 3500. At step 3705, the processor of the hearing aidsystem may selectively condition the received audio data using anysuitable approaches described above. For example, the processor mayselectively condition the received audio data by analyzing the data andseparating one or more voice audio data related to the one or morespeakers from the received audio data. In some embodiments, selectivelyconditioning the audio data may include removing audio background noisedata as described above.

At step 3707, the conditioned audio signal may be provided, for exampletransmitted, to the hearing aid system of user 100. For example, theconditioned audio signal may be provided to a hearing interface device(e.g., earpiece, headphones, speaker, etc.). Step 3707 may be similar tostep 3524 of process 3500.

FIG. 37B shows an illustrative process 3760 of transmitting aconditioned audio signal to a device (e.g., an earpiece of the hearingaid system). At step 3761, a processor of the hearing aid system may beprogrammed to receive an audio signal from the at least one microphone.In various embodiments, step 3761 may be the same as step 3602 ofprocess 3500.

At step 3762, a processor of the hearing aid system may determine if theaudio signal is associated with or comprises speech by a recognizedindividual. The processor may make the determination using any of thesuitable approaches described above, such as comparing the audio signalfrom the individual with a voiceprint of a person whose image is beingrecognized, using a trained engine, or the like.

At step 3763, an image of the individual may be displayed on a screen ofa computing device (e.g., a mobile device) available for user 100. Invarious embodiments, step 3763 may be the same as step 3514 of process3500. At step 3764, the processor of the hearing aid system mayselectively condition the received audio signal. In various embodiments,step 3764 may be the same as step 3520 of process 3500. In someembodiments, the audio signal may be transmitted to server 250, and aprocessor of server 250 may selectively condition the audio signal.Selective conditioning of the received audio signal may be achievedusing any suitable approaches described above. For example, theprocessor of the hearing aid system may selectively condition thereceived audio signal by analyzing the signal and separating one or morevoice audio data related to the one or more speaker from the receivedaudio signal. In some embodiments, selectively conditioning the audiosignal may include removing audio background noise data as describedabove.

At step 3766, the conditioned audio signal may be provided to a hearingaid interface of user 100. For example, the conditioned audio signal maybe transmitted to a hearing interface device (e.g., earpiece,headphones, speaker, etc.). Step 3766 may be similar to step 3524 ofprocess 3500.

FIG. 37C shows an illustrative process 3770 of transmitting aconditioned audio signal to a device (e.g., an earpiece of the hearingaid system). At step 3771, a processor of the hearing aid system may beprogrammed to receive an audio signal from the at least one microphone.In various embodiments, step 3771 may be the same as step 3516? ofprocess 3500.

At step 3772, a processor of the hearing aid system may detect, based onanalysis of the audio signals, a first audio signal associated with afirst time period, wherein the first audio signal is representative of avoice of a single individual. In an example embodiment, the first audiosignal may correspond to a single person communicating with user 100during the first time window. Alternatively, during a first time window,multiple individuals may be communicated with user 100, having distinctvoices, that the processor of the hearing aid system is capable ofseparating using any of the suitable approaches discussed above. In someembodiments, the audio signal associated with the first time window maybe transmitted to server 250, and a processor of server 250 may separateaudio signal to extract voices of individuals communicating with user100.

At step 3773, the processor of the hearing aid system may detect, basedon analysis of the audio signals, a second audio signal associated witha second time period, wherein the second time period is different fromthe first time period, and wherein the second audio signal isrepresentative of overlapping voices of two or more individuals. Forexample, the second time window may correspond to an instance whenmultiple speakers are talking at the same time. In some embodiments, thefirst or the second time window may not have to correspond to acontinuous time interval. For example, during a conversation, a speechof a single individual may be overlapped at various times by the voicesof other individuals. In such cases, the time when a single individualis speaking corresponds to the first time window and the time whenmultiple individuals are speaking correspond to the second time window

At step 3774, the first detected audio signal may be selectivelyconditioned using any of the suitable approaches discussed above. Invarious embodiments, step 3774 may be similar to step 3520 of process3500. At step 3775, the second detected audio signal may be selectivelyconditioned using any of the suitable approaches discussed above. Invarious embodiments, the selective conditioning of the first audiosignal may be different in at least one respect relative to theselective conditioning of the second audio signal. For example, thefirst audio signal may be amplified while the second audio signal may besuppressed. In various embodiments, some of the voices presented in thesecond audio signal may be separated and suppressed, and other voicespresented in the second audio signal may be amplified, or modified inany suitable way as discussed above.

At step 3776, the conditioned audio signal may be provided to thehearing aid system of user 100. For example, the conditioned audiosignal may be transmitted to a hearing interface device (e.g., earpiece,headphones, speaker, etc.). Step 3776 may be similar to step 3524 ofprocess 3500.

In various embodiments, the processor of the hearing aid system may beprogrammed to analyze one or more images captured by a wearable cameraof the hearing aid system and identify two or more individuals, whereinthe selective conditioning of the first audio signal and the secondaudio signal is based on information associated with an identity of atleast one of the two or more individuals. For example, if it isimportant for user 100 to clearly hear the voice of the identifiedindividual (e.g., when one of the individuals is the user's boss) thevoice of the identified individual may be amplified. In variousembodiments, one of the two individuals may be identified using any ofthe suitable approaches discussed above. For example, the individual maybe identified using a computer-based model trained to recognize peoplewithin images. In some cases, the individual may also be identifiedbased on an audio signal detected within the first or the second audiosignal. For example, the hearing aid system may retrieve from thedatabase of server 250 various voiceprints of known individuals and useone or more of the retrieved voiceprints to identify a voice of a knownindividual within the first or the second audio signal.

In some embodiments, the hearing aid system may be configured tocondition the first and the second audio signal and modify voicesassociated with identified individuals in any suitable way as discussedabove. For example, the hearing aid system may suppress one or morevoices associated with the one or more identified individual, amplifythe one or more voices, change the pitch or the rate of the one or morevoices, and the like. As another example, when two individuals arepresent, and one individual is identified, the voice of the identifiedindividual may be amplified, and the voice of the second individual maybe suppressed. As yet another example, the voice of the secondindividual may be transcribed and displayed on a device associated withthe hearing aid system of user 100.

In various embodiments, when the audio signal contains overlappingvoices of various individuals, the hearing aid system, may identifyaudio signals related to the voices, and selectively condition the audiosignal (e.g., by amplifying and suppressing voices) using any suitablelogic. For example, the hearing aid system may amplify voices related toa particular topic, voices, of individuals engaging in conversation withuser 100, voices of individuals engaging in conversation with aparticular individual, voices of individuals selected by user 100, andthe like. In some cases, the hearing aid system may be configured tosuppress the overlapping voices of a background conversation (e.g., aconversation between various speakers that are not directly conversingwith user 100). In some cases, the hearing aid system may suppress thevoices that cannot be transcribed by the hearing aid system (e.g., thevoices that are cannot be clearly heard, or voices that present nodiscernable useful information, such as voices that produce sounds thatcan be interpreted to correspond to words).

Identifying Information and Associated Individuals

According to embodiments of this disclosure, a hearing aid system mayrecognize speakers in a surrounding environment of a user (e.g., user100 in FIGS. 1A-1B and 2). In some embodiments, the hearing aid systemmay further recognize that one speaker is talking to another speaker oruser 100. Such recognition may be implemented through image analysis,audio analysis, or both. The hearing aid system may transcriberecognized conversations between the speakers. In some embodiments, theconversations may be associated with respective identifiers (e.g.,names) of the speakers, if the speakers are recognized individuals. Insome embodiments, the hearing aid system may capture instructions oraction items directed to user 100 from a speaker (e.g., when user 100 isin a meeting).

FIG. 38A is a block diagram illustrating a hearing aid system 3800according to an example embodiment. As shown in FIG. 38A, the hearingaid system 3800 includes at least one wearable camera 3801, at least onemicrophone 3802, at least one processor 3803, and a memory 3804. In someembodiments, system 3800 may further include other components, such ascomponents as shown in FIGS. 5A-5C.

In some embodiments, wearable camera 3801 may capture images from anenvironment of user 100. In some embodiments, wearable camera 3801 mayinclude image sensor 220 in FIG. 5A or 5C. In some embodiments, wearablecamera 3801 may include at least one of image sensors 220 a or 220 b inFIG. 5B.

In some embodiments, microphone 3802 may capture sounds from theenvironment of user 100. In some embodiments, microphone 3802 mayinclude a directional microphone. In some embodiments, microphone 3802may include multiple microphones (e.g., a microphone array). In suchcases, one microphone may capture only the background noise, whileanother microphone may capture a combined audio including the backgroundnoise as well as individuals' voices. Processor 3803 may obtain thevoices by subtracting the background noise from the combined audio. Insome other embodiments, system 3800 may include at least one microphoneand a pressure sensor (not shown). The pressure sensor may encode airpressure differences (e.g., caused by a sound wave) as a digital signal.System 3800 may process the sounds captured by microphone 3802 and thedigital signal captured by the pressure sensor to separate the requiredvoice and the background noise.

In some embodiments, wearable camera 3801 and microphone 3802 may beincluded in a common housing (e.g., a shell). For example, wearablecamera 3801 and microphone 3802 may be included in a common housing ofapparatus 110 in FIGS. 3A-3B and 4A-4B.

In some embodiments, processor 3803 may be implemented as processor 210in FIG. 5A or 5C. In some embodiments, processor 3803 may be implementedas at least one of processors 210 a or 210 b in FIG. 5B. In someembodiments, processor 3803 may be implemented as processor 540 in FIG.5C. In some embodiments, processor 3803 may be included in the commonhousing that includes wearable camera 3801 and microphone 3802. Forexample, processor 3803 may be included in the common housing ofapparatus 110 in FIGS. 3A-3B and 4A-4B. In some embodiments, processor3803 may be included in a second housing separate from the commonhousing. In some embodiments, the second housing may be associated witha paired mobile device. For example, the mobile device may be computingdevice 120 in FIG. 1A-1D, 2, or 5C. The mobile device may be paired withsystem 300 via, for example, a wireless link (e.g., Bluetooth® link). Insuch a case, processor 3803 (e.g., implemented as processor 540 in FIG.5C) may be included in a housing of computing device 120. When processor3803 is in the second housing, in some embodiments, processor 3093 mayreceive data (e.g., the images captured by wearable camera 3801) via awireless link between a transmitter (e.g., wireless transceiver 503 a inFIG. 5C) in the common housing and receiver (e.g., wireless transceiver503 b in FIG. 5C) in the second housing. For example, the wireless linkmay be a Bluetooth® link, a Wi-Fi link, a near-field communications(NFC) link, or the like.

In some embodiments, memory 3804 may be implemented as memory 550 asshown in FIGS. 5A and 5B. In some embodiments, memory 3804 may beimplemented as at least one of memories 550 a and 550 b in FIG. 5C.

Apparatus 110 may be configured to deduce instructions from anindividual in the environment of user 100. FIG. 38B is a schematicillustration showing an exemplary environment for use of a hearing aidwith instruction deduction capabilities consistent with the presentdisclosure.

As shown, apparatus 110 may be configured to recognize a face 3805 orvoice 3806 associated with an individual 3807 within the environment ofuser 100. For example, apparatus 110 may be configured to capture one ormore images of the surrounding environment of user 100 using wearablecamera 3801. The captured images may include a representation of arecognized individual 3807, which may be a friend, colleague, relative,or prior acquaintance of user 100. Processor 3803 (e.g., processors 210a and/or 210 b) may be configured to analyze the captured images anddetect the recognized user using various facial recognition techniques.Accordingly, apparatus 110, or specifically memory 550, may comprise oneor more facial or voice recognition components.

Processor 3803 may further be configured to determine whether individual3807 is recognized by user 100 based on one or more detected audiocharacteristics of sounds associated with a voice of individual 3807.Processor 3803 may determine that sound 3808 corresponds to voice 3806of user 3807. Processor 3803 may analyze audio signals representative ofsound 3808 captured by microphone 3802 to determine whether individual3807 is recognized by user 100. This may be performed using one or morevoice recognition algorithms, such as Hidden Markov Models, Dynamic TimeWarping, neural networks, or other techniques. Voice recognitioncomponent and/or processor 3803 may access a database (not shown), whichmay further include a voiceprint of one or more individuals. Processor3803 may perform voice recognition to analyze the audio signalrepresentative of sound 3808 to determine whether voice 3806 matches avoiceprint of an individual in the database. Accordingly, the databasemay contain voiceprint data associated with a number of individuals.After determining a match, individual 3807 may be determined to be arecognized individual of user 100. This process may be used alone, or inconjunction with the facial recognition techniques. For example,individual 3807 may be recognized using facial recognition and may beverified using voice recognition, or vice versa.

After determining that individual 3807 is a recognized individual ofapparatus 110, processor 3803 may cause selective conditioning of audioassociated with the recognized individual. The conditioned audio signalmay be transmitted to a hearing interface device (e.g., a speaker or anearphone), and thus may provide user 100 with audio conditioned based onthe recognized individual. For example, the conditioning may includeamplifying audio signals determined to correspond to sound 3808 (whichmay correspond to voice 3806 of user 3807) relative to other audiosignals. In some embodiments, amplification may be accomplisheddigitally, for example by processing audio signals associated with sound3808 relative to other signals. Additionally, or alternatively,amplification may be accomplished by changing one or more parameters ofmicrophone 3802 to focus on audio sounds associated with individual3807. For example, microphone 3802 may be a directional microphone andprocessor 3803 may perform an operation to focus microphone 3802 onsound 3808. Various other techniques for amplifying sound 3808 may beused, such as using a beamforming microphone array, acoustic telescopetechniques, etc.

In some embodiments, selective conditioning may include attenuation orsuppressing one or more audio signals received from directions notassociated with individual 3807. For example, processor 3803 mayattenuate sounds 3809 and 3810. Similar to amplification of sound 3808,attenuation of sounds may occur through processing audio signals, or byvarying one or more parameters associated with microphone 3802 to directfocus away from sounds not associated with individual 3807.

Selective conditioning may further include determining whetherindividual 3807 is speaking. For example, processor 3803 may beconfigured to analyze images or videos containing representations ofindividual 3807 to determine when individual 3807 is speaking, forexample, based on detected movement of the recognized individual's lips.This may also be determined through analysis of audio signals receivedby microphone 3802, for example by detecting the voice 3806 ofindividual 3807. In some embodiments, the selective conditioning mayoccur dynamically (initiated and/or terminated) based on whether or notthe recognized individual is speaking.

In some embodiments, conditioning may further include changing a tone ofone or more audio signals corresponding to sound 3808 to make the soundmore perceptible to user 100. For example, user 100 may have lessersensitivity to tones in a certain range and conditioning of the audiosignals may adjust the pitch of sound 3808. In some embodimentsprocessor 3803 may be configured to change a rate of speech associatedwith one or more audio signals. For example, sound 3808 may bedetermined to correspond to voice 3806 of individual 3807. Processor3803 may be configured to vary the rate of speech of individual 3807 tomake the detected speech more perceptible to user 100. Various otherprocessing may be performed, such as modifying the tone of sound 3808 tomaintain the same pitch as the original audio signal, or to reduce noisewithin the audio signal.

In some embodiments, processor 3803 may determine a region 3811associated with individual 3807. Region 3811 may be associated with adirection of individual 3807 relative to apparatus 110 or user 100. Thedirection of individual 3807 may be determined using wearable camera3801 and/or microphone 3802 using the methods described above. As shownin FIG. 38B, region 3811 may be defined by a cone or range of directionsbased on a determined direction of individual 3807. The range of anglesmay be defined by an angle, θ, as shown in FIG. 38B. The angle, θ, maybe any suitable angle for defining a range for conditioning soundswithin the environment of user 100 (e.g., 10 degrees, 20 degrees, 45degrees). Region 3811 may be dynamically calculated as the position ofindividual 3807 changes relative to apparatus 110. For example, as user100 turns, or if individual 3807 moves within the environment, processor3803 may be configured to track individual 3807 within the environmentand dynamically update region 3811. Region 3811 may be used forselective conditioning, for example by amplifying sounds associated withregion 3811 and/or attenuating sounds determined to be emanating fromoutside of region 3811.

The conditioned audio signal may then be transmitted to the hearinginterface device and produced for user 100. Thus, in the conditionedaudio signal, sound 3808 (and specifically voice 3806) may be louderand/or more easily distinguishable than sounds 3809 and 3810, which mayrepresent background noise within the environment.

In some embodiments, processor 3803 may determine the direction of arecognized individual relative to the user based on the images. In someembodiments, processor 3803 may be configured to determine a lookdirection of the individuals in the images. In some embodiments, whenthe recognized individual is speaking to the user, the selectiveconditioning may include amplifying an audio signal associated with therecognized individual relative to other audio signals received fromdirections outside a region associated with the recognized individual.If the recognized individual is speaking towards the user (e.g.,individual 3807 speaking towards user 100 in FIG. 38B), processor 3803may transcribe text corresponding to speech associated with the voice ofthe recognized individual.

FIG. 38C illustrates a user wearing an exemplary hearing aid system.User 100 may wear system 3800 (e.g., as a wearable device). Wearablecamera 3801 may capture images of the environment of user 100. Asillustrated in FIG. 38C, a first individual 3812 may stand in front ofuser 100 and look in the direction of user 100. In addition, a secondindividual 3813 may also stand in front of user 100, but look in adirection away from user 100. The image sensor of system 3800 maycapture one or more images including first individual 3812 and secondindividual 3813. Processor 3803 may analyze the images captured bywearable camera 3801. Processor 3803 may also identify one or moreindividuals included in the images, based on image analysis or facerecognition. For example, processor 3803 may identify first individual3812 and second individual 3813 included in the image. Based on theanalysis, processor 3803 may detect that first individual 3812 islooking in the direction of user 100 and second individual 3813 islooking in a direction away from user 100. Microphone 3804 may receiveone or more audio signals from the environment of user 100. For example,microphone 3804 may be configured to receive (or detect) a first audiosignal associated with the voice of the first individual 3812 and asecond audio signal associated with the voice of the second individual3813. In the example as shown in FIG. 38C, based on the lookingdirections of first individual 3812 and second individual 3813,processor 3803 may transcribe text corresponding to speech associatedwith the voice of first individual 3812, but not text corresponding tospeech associated with the voice of second individual 3813.

In some embodiments, processor 3803 may be programmed to perform amethod for deducing instructions for user 100. FIG. 39A is a flowchartillustrating a process 3900A for deducing instructions for a hearing aidsystem according to an embodiment. Processor 3803 may perform process3900A to recognize an individual in a surrounding environment of user100 after system 300 captures the voices or images of the individual.

At step 3902, processor 3803 may receive the images captured by wearablecamera 3801. In some embodiments, the images may include human beings.In some embodiments, wearable camera 3801 may capture the imagessubstantially in line with a line of sight of user 100, such that anindividual user 100 is speaking with may be likely to be at or near thecenter of the images.

At step 3904, processor 3803 receives audio signals representative ofsounds captured by microphone 3802. In some embodiments, the audiosignals may include speech or non-speech sounds by one or more personsin the vicinity of user 100, environmental sound (e.g., music, tones, orenvironmental noise), or the like. In some embodiments, the sounds maybe an audio stream. The audio stream may be made up of a combination ofaudio signal components. Each of the audio signal components may beseparated to provide a unique audio signal. Processor 3803 may thenreceive a plurality of such unique audio signals.

At step 3906, processor 3803 may identify a first individual representedin at least one of the images. In some embodiments, processor 3803 mayreceive a plurality of images. In some embodiments, step 3906 may beoptional. The first individual may appear in some or all of theplurality of images. In some embodiments, processor 3803 may implementan image processing technique (e.g., an algorithm or a software module)to recognize individuals in the images. Such an image processingtechnique may be based on geometry. For example, processor 3803 mayidentify an individual at the center of the image as the firstindividual. For example, processor 3803 may identify a chin of the userin the images and then identify another individual opposite the user.

In some embodiments, processor 3803 may amplify the first audio signal.For example, processor 3803 may amplify the first audio signal bychanging tones or applying a noise cancellation technique (e.g., analgorithm or a software module). In some embodiments, processor 3803 maycause transmission (e.g., using wireless transceiver 530 or 530 a inFIGS. 5A-5C) of the amplified first audio signal to a hearing interfacedevice that is configured to provide sound to an ear of user 100. Byproviding the sound of the amplified first audio signal, user 100 may beable to concentrate on the first individual with fewer disturbances ofother voices or sounds. For example, the hearing interface device mayinclude a speaker associated with an earpiece. For another example, thehearing interface device may include a bone conduction microphone.

In some embodiments, processor 3803 may transmit the amplified firstaudio signal as long as the first individual keeps speaking. Processor3803 may transmit the amplified first audio even if other voices orsounds are captured by microphone 3802, whether recognized or not, inorder to let user 100 continuously listen to the first individual. Insome embodiments, when the first individual pauses for up to apredetermined length, processor 3803 may determine it as an end of aspeech by the first individual and attempt to detect speech of otherindividuals. In some embodiments, processor 3803 may amplify audiosignals of other individuals to a different degree from the first audiosignal.

Referring back to FIG. 39A, at step 3908, processor 3803 may identify afirst audio signal. The first audio signal may be representative of avoice of the first individual from among the received audio signals.However, the first audio signal may also be associated with another, orunknown, speaker. In some embodiments, the first audio signal may bepreprocessed to be separated from background noise captured bymicrophone 3802.

At step 3910, processor 3803 may transcribe and store, in memory 3804,text corresponding to the speech, which may be associated with the voiceof the first individual, if the individual has been associated with thespeech. In some embodiments, the voice may include speech (e.g., aconversation or a verbal instruction). The voice may further includenon-speech sounds (e.g., laughter, crying, or noise). Processor 3803 mayimplement a text-to-speech technique (e.g., a text-to-speech algorithmor software module) to recognize the speech from the voice andtranscribe the speech associated with the voice into the text.

At step 3912, processor 3803 may determine whether the speaker is thefirst individual, and whether the first individual is a recognizedindividual. In some embodiment, processor 3803 may recognize the firstindividual by analyzing the first audio signal associated. For example,the voice of the first individual may have been previously recognized(e.g., in a different conversation or in an earlier part of the sameconversation), and features (e.g., vocal prints) of the recognized voiceof the first individual may be stored in memory 3804 (e.g., in adatabase). When processor 3804 analyzes the first audio signal, it maydetermine the features of the first audio signal (e.g., by extractingvocal prints) and search memory 3804 (e.g., in the database) to seek amatch. If such a match is found, processor 3803 may determine thematching between the speaker and the first individual, and that thefirst individual is a recognized individual.

In some embodiments, processor 3803 may recognize the first individualbased on imaged facial features extracted from the at least one imageidentified at step 3803. For example, the images of the first individualmay have been previously recognized (e.g., using a facial recognitionalgorithm), and the first individual's facial features may be stored inmemory 3804 (e.g., in a database). When processor 3804 analyzes theidentified images, it may determine the facial features of the firstindividual and search memory 3804 (e.g., in the database) to seek amatch. If such a match is found, processor 3803 may determine that thefirst individual is a recognized individual.

In some embodiments, processor 3803 may recognize the first individualbased on both the first audio signal and the identified images. Itshould be noted that processor 3803 may also use other methods,processes, algorithms, or means to recognize the first individual, notlimited to the examples as described herein.

Referring back to FIG. 39A, at step 3914, if the first individual is arecognized individual, processor 3803 may associate an identifier of thefirst recognized individual with the stored text corresponding to thespeech associated with the voice of the first individual. In someembodiments, for example, if the text is stored in a database (e.g., arelational database) in memory 3804, processor 3803 may add or change arecord in the relational database to store the identifier as a key ofthe text, in which the text is stored as a value. In some embodiments,processor 3803 may prompt user 100 for identifying information relatingto the first individual if the first individual is not a recognizedindividual. For example, processor 3803 may prompt user 100 to speak anidentifier (e.g., a name, a label, or a tag) of the first individual.For another example, processor 3803 may prompt user 100 an input fieldin a user interface (e.g., on display 260 in FIG. 5C) for user 100 toinput the identifier of the first individual.

FIG. 39B is a flowchart illustrating a process 3900B for deducinginstructions for a hearing aid system according to an embodiment.Process 3900B may follow step 3910 of process 3900A. Processor 3803 mayperform process 3900B to recognize whether the individual is speakingtowards user 100.

At step 3916, processor 3803 may determine whether the speech associatedwith the voice of the first individual is directed toward user 100. Insome embodiments, processor 3803 may determine whether the speechassociated with the voice of the first individual is directed towarduser 100 based on at least one of a detected look direction of user 100or a detected look direction of the first individual. For example,processor 3803 may determine the look direction of user 100 based ondetection of a chin of user 100 in at least one of the images. Foranother example, processor 3803 may determine the look direction of thefirst individual based on detection of one or more eyes of the firstindividual in at least one of the images and based on at least onecharacteristic of the one or more detected eyes. For another example,processor 3803 may determine the look direction of the first individualbased on gestures, gaits, or body movement features of the firstindividual detected from at least one of the images. For anotherexample, processor 3803 may determine the look direction of the firstindividual based on the user's name included in the speech of the firstindividual.

At step 3918, if the speech of the first individual is directed towarduser 100, processor 3803 may store in memory 3804 an indication that thefirst individual's speech is directed toward user 100. For example,processor 3803 may store the indication in the relational databasedescribed at step 3914.

FIG. 40A is a flowchart illustrating a process 4000A for deducinginstructions for a hearing aid system according to an embodiment. Insome embodiments, process 4000A may follow any step of process 3900A or3900B. Processor 3803 may implement process 4000A to recognize multipleindividuals in the surrounding environment of user 100 and transcribetheir speech if the voice or images of the individual is captured bysystem 300.

At step 4002, processor 3803 may identify a second individualrepresented in at least one of the images. Step 4002 may be implementedin a manner similar to step 3906. For example, processor 3803 may usethe image processing algorithm to identify individuals in images basedon the individuals' traits or characteristics of at least one of thebody shapes, motions, or facial expressions. If recognized individualshave different traits or characteristics, processor 3803 may determinethat the second individual is identified in the images.

At step 4004, processor 3803 may identify a second audio signal, fromamong the received audio signals, representative of a voice of thesecond individual. Step 4004 may be implemented in a manner similar tostep 3908. For example, processor 3803 may extract features (e.g., vocalprints) from audio signals. If the extracted features are not the same,processor 3803 may determine that the second signal is identified in thereceived audio signals.

At step 4006, processor 3803 may transcribe and store, in memory 3804,text corresponding to speech associated with the voice of the secondindividual. Step 4006 may be implemented in a manner similar to step3910.

FIG. 40B is a flowchart illustrating a process 4000B for deducinginstructions for a hearing aid system according to an embodiment. Insome embodiments, process 4000B may follow step 4006 of process 4000A.Processor 3803 may implement process 4000B to recognize that whether thesecond individual is a recognized individual. Processor 3803 mayimplement process 4000B to further recognize whether the secondindividual is speaking towards user 100, whether the second individualis speaking towards the first individual, or whether the firstindividual is speaking towards the second individual.

At step 4008, processor 3803 may determine whether the second individualis a recognized individual. Step 4008 may be implemented in a mannersimilar to step 3912.

At step 4010, if the second individual is a recognized individual,processor 3803 may associate an identifier of the second recognizedindividual with the stored text corresponding to speech associated withthe voice of the second individual. Step 4010 may be implemented in amanner similar to step 3914.

At step 4012, processor 3803 may determine whether the speech associatedwith the voice of the second individual is directed toward user 100.Step 4012 may be implemented in a manner similar to step 3916.

At step 4014, if the speech of the second individual is directed towarduser 100, processor 3803 may store in memory 3804 an indication that thesecond individual's speech is directed toward user 100. Step 4014 may beimplemented in a manner similar to step 3918.

At step 4016, processor 3803 may determine whether the speech associatedwith the voice of the second individual is directed toward the firstindividual. Step 4016 may be implemented in a manner similar to step3916 or 4012. In some embodiments, processor 3803 may determine whetherthe speech associated with the voice of the second individual isdirected toward the first individual based on a look direction of thesecond individual detected based on analysis of at least one of theimages. In some embodiments, processor 3803 may determine whether thespeech associated with the voice of the second individual is directedtoward the first individual based on detection of a name associated withthe first individual in the speech of the second individual.

At step 4018, if the speech of the second individual is directed towardthe first individual, processor 3803 may store in memory 3804 anindication that the second individual's speech is directed toward thefirst individual. Step 4018 may be implemented in a manner similar tostep 3918 or 4014.

At step 4020, processor 3803 may determine whether the speech associatedwith the voice of the first individual is directed toward the secondindividual. Step 4016 may be implemented in a manner similar to step3916, 4012, or 4016.

At step 4022, if the speech of the first individual is directed towardthe second individual, processor 3803 may store in memory 3804 anindication that the first individual's speech is directed toward thesecond individual. Step 4022 may be implemented in a manner similar tostep 3918, 4014, or 4018.

In some embodiments, processes 39A-39B or 40A-40B may include additionalsteps. For example, processor 3803 may perform those additional stepsafter any step of 39A-39B or 40A-40B.

In some embodiments, processor 3803 may cause the stored text (e.g., atstep 3910 or 4006) to be shown on a display. In some embodiments, thedisplay (not shown in FIG. 38A) may be included in the common housingthat includes wearable camera 3801 and microphone 3802. In someembodiments, the display may be associated with a paired mobile devicepaired with system 300. For example, the mobile device may be computingdevice 120 in FIG. 1A-1D, 2 or 5C. The display may be, for example,display 260 in FIG. 5C.

In some embodiments, processor 3803 may generate a task item based onanalysis of the speech associated with the voice of the firstindividual. In some embodiments, processor 3803 may implement atask-context matching technique (e.g., an algorithm or a softwaremodule) to determine whether the context (e.g., the stored text of therecognized individual's speech) is suitable for any task (e.g., anundated task or a dated task). For example, processor 3803 may implementthe task-context matching technique to recognize that the context of thespeech between the recognized individuals and user 100 is a meeting.Based on that context, processor 3803 may further determine that thecontext is suitable for receiving tasks. Based on the stored text andthe direction of the speech (e.g., as recognized in process 400B),processor 3803 may determine the content of the task. In someembodiments, processor 3803 may implement a suggestion technique (e.g.,an algorithm or a software module) to suggest the task to user 100. Insome embodiments, the suggestion technique may include a naturallanguage processing technique. For example, processor 3803 may cause tosuggest performing the task or setting a date for the task. In someembodiments, processor 3803 may identify tasks by specific words in thespeech, such as “please”, prepare”, “e-mail me”, “send”, or the like. Insome embodiments, processor 3803 may attach a due date to a task, basedon identified times or dates in the speech, such as, for example, “byWednesday noon”, “next week”, or the like.

In some embodiments, processor 3803 may deduce or identify instructionsbased on at least one of the speech, images, or the transcribed texts.For example, processor 3803 may analyze the context of the speech of thefirst individual using natural language processing technique todetermine whether there is an instruction included. For another example,processor 3803 may analyze the images captured and analyze gestures,gaits, facial expressions, body movements to determine whether there isan instruction included, such as nodding a head, shaking a head, raisinga hand, or the like. For another example, processor 3803 may analyze thetranscribed texts to determine whether there is an instruction included,such as generating a task item as previously described. In someembodiments, processor 3803 may determine whether there is aninstruction from the first individual using a combination of any of thespeech, images, or the transcribed texts. In some embodiments, processor3803 may determine whether the recognized instruction is directed to theuser or to a second individual based on who the first individual isspeaking to. In some embodiments, processor 3803 may further check withcontextual information to further determine whether there is aninstruction included. For example, when processor 3803 recognized acandidate instruction related to adding an item to a schedule orcalendar of the user, processor 3803 may checked the schedule orcalendar for conflicts to determined that whether the item is restatedor newly added.

In some embodiments, processor 3803 may update a database associatedwith user 100 to include the generated task item. For example, thedatabase may be the database in memory 3802 as described in step 3912,3914, or 3918. Processor 3803 may store or update the generated taskitem as a data record in the database, for example.

In some embodiments, processor 3803 may collect the tasks throughout theday and provide them to the user upon request.

Selectively Conditioning of Audio Signals Based on an Audioprint of anObject

Human beings have distinct and different voices. While some people havea good voice memory and can easily recognize their first primary schoolteacher, other people may have difficulty recognizing their closestfriends only from their voice. Nowadays, computer algorithms surpassmost people in recognizing speakers because they can identify anddistinguish the human voice. The way these machine-learning algorithmsrecognize speakers is based on mathematical solutions that useaudioprints. The term “audioprint,” also known as “acoustic fingerprint”and “voice signature,” refers to a condensed digital summary of thespecific acoustic features of a sound-emanating object (e.g.,individuals and also inanimate objects) deterministically generated froma reference audio signal. A common technique for determining anaudioprint from recorded audio signals is using a time-frequency graphcalled a spectrogram. For example, the disclosed hearing aid system mayidentify in the spectrogram multiple points (e.g., peak intensitypoints) related to different words or vocal sounds created by anindividual talking to user 100. The disclosed hearing aid system mayaccess multiple reference audioprints associated with differentsound-emanating objects stored in a local or a cloud-based database.Using the reference audioprints, the disclosed hearing aid system maydetermine an audioprint from recorded audio signals and identify thesound-emanating object responsible for generating the audio signals.Consistent with the present disclosure, the hearing aid system mayretrieve information relating to the identified sound-emanating objectand cause selective conditioning of at least one audio signal associatedwith the identified sound-emanating object based on the retrievedinformation. For example, when user 100 is at the park with his child,the disclosed system may amplify the voice of the child relative to thevoices of other nearby children.

FIG. 41A illustrates an exemplary embodiment of a memory 4100 containingsoftware modules consistent with the present disclosure. In particular,as shown, memory 4100 may include an audio analysis module 4102, anaudioprint determination module 4104, a database access module 4106, aselective conditioning module 4108, a transmission module 4110, and adatabase 4112. Modules 4102, 4104, 4106, 4108, and 4110 may containsoftware instructions for execution by at least one processing device(e.g., processor 210, included with the suggested hearing aid system).Audio analysis module 4102, audioprint determination module 4104,database access module 4106, selective conditioning module 4108,transmission module 4110, and database 4112 may cooperate to performmultiple operations.

For example, the hearing aid system may be used to selectively conditionaudio signals based on a determined audioprint of a sound-emittingobject. For example, audio analysis module 4102 may receive audiosignals representative of sounds emanating from objects in anenvironment of user 100 and analyze the received audio signals to obtainan isolated audio stream associated with one sound-emanating object.Audioprint determination module 4104 may determine an audioprint of thesound-emanating object from the isolated audio stream. In oneimplementation, audioprint determination module 4104 may use deeplearning algorithms or neural embedding models to determine theaudioprint of the sound-emanating object. Database access module 4106may interact with database 4112, which may store information relating tosound-emanating objects associated with user 100 and any otherinformation associated with the functions of modules 4102-4110. Forexample, database access module 4106 may use the determined audioprintto retrieve information relating to a detected sound-emanating objectfrom database 4112. The retrieved information may include relationshiplevel indicators between user 100 and the detected sound-emanatingobject, or specific audio conditioning rules associated with anidentified sound-emanating object. Selective conditioning module 4108may cause selective conditioning of at least one audio signal associatedwith the identified sound-emanating object. For example, selectiveconditioning module 4108 may amplify sounds from the user's smartphoneand avoid from amplifying sounds from other phones. Transmission module4110 may cause transmission of the at least one conditioned audio signalto a hearing interface device (e.g., hearing interface device 1710)configured to provide sounds to an ear of user 100.

In another example, the hearing aid system may attenuate backgroundnoise based on determined audioprints of objects in the environment ofuser 100. For example, audio analysis module 4102 may receive audiosignals representative of sounds from the environment of user 100 andanalyze the received audio signals to isolate a plurality of audiostreams associated with a corresponding plurality of sound-emanatingobjects in the environment of user 100. Each sound-emanating object inthe environment of user 100 may be associated with a unique audioprint.Audioprint determination module 4104 may determine a plurality ofaudioprints associated with the plurality of isolated audio streams.Database access module 4106 may use the determined audioprints to obtainat least one indicator of a type associated with each of the pluralityof sound-emanating objects. The at least one indicator may be indicativeof a level of interest user 100 has with the type associated with eachof the plurality of sound-emanating objects. Selective conditioningmodule 4108 may cause selective conditioning of the plurality ofisolated audio streams based on the determined at least one indicator ofa type associated with each of the plurality of sound-emanating objects.For example, selective conditioning module 4108 may selectivelyattenuate a first audio stream determined to be associated withbackground noise relative to a second audio stream determined to not beassociated with background noise. Transmission module 4110 may causetransmission of the conditioned audio signals (and the non-conditionedaudio signals) to a hearing interface device configured to providesounds to an ear of user 100. Additional details on this exampleoperation are provided below with reference to FIGS. 44A-44C.

Consistent with embodiments of the present disclosure, memory 4100 mayalso include an object identification module (not shown). The objectidentification module may identify the at least one sound-emanatingobject in the environment of user 100 based on the audioprint determinedby audioprint determination module 4104. Moreover, in an embodiment, theobject identification module may identify the at least onesound-emanating object in the environment of user 100 using imageanalysis. For example, the object identification module may receive aplurality of images depicting one or more sound-emanating objects. Theplurality of images may be captured by a wearable camera located in asame housing that includes the wearable microphone (e.g., apparatus110). According to this embodiment, the object identification module maydetermine visual characteristics of a sound-emanating object based onanalysis of the plurality of images. Thereafter, the objectidentification module may use the determined visual characteristics andthe determined audioprint to identify the sound-emanating object. Forexample, the visual characteristics of the sound-emanating object mayinclude facial features of an individual speaking with user 100.Database access module 4106 may retrieve from database 4112 predefinedsettings associated with an identity of the sound-emanating object.Thereafter, selective conditioning module 4108 may cause theconditioning of at least one audio signal based on the identity of thesound-emanating object. For example, sounds generated by specificsound-emitting objects may be silenced (e.g., the intensity of theconditioned audio signals associated with the AC may be 0%) and soundsgenerated by other specific sound-emitting objects may be amplified(e.g., the intensity of the conditioned audio signals associated with afamily member may be 110%).

After the object identification module identifies the sound-emanatingobject (e.g., using audio analysis, using image analysis, or using acombination of both), database access module 4106 may retrievepredefined settings related to the identified sound-emanating object. Inone embodiment, the predefined settings may be associated withmodifications to audio signals generated by the sound-emanating object.For example, according to one predefined setting, audio signals from aspecific sound-emanating object may be silenced whenever encountered. Inanother embodiment, the predefined settings may be defined by or bespecific to user 100. For example, Alice may want to amplify soundsgenerated by babies and Bob may want to silence sounds generated bybabies. In another embodiment, the predefined settings may be contextrelated. In other words, a single sound-emanating object may beassociated with different settings that correspond with differentsituations. Thus, the hearing aid system may identify the situation user100 is in and apply the settings associated with sound-emanating objectsaccordingly. As another example, when Alice is at home, she may want toamplify sounds generated by babies; but when Alice is at work, she maywant to silence sounds generated by babies. The hearing aid system maydetermine if Alice is at home or at work (e.g., through analysis ofimages captured of Alice's surrounds, by accessing informationassociated with a calendar or schedule, and/or via accessing GPS orother location information) and modify the audio signals associated withbabies accordingly.

Consistent with additional embodiments of the present disclosure, thesoftware modules illustrated in FIG. 41A may be stored in separatememory devices. In one example embodiment, selective conditioning module4108 may be stored in a memory device located in a hearing interfacedevice (e.g., hearing interface device 1710). The hearing interfacedevice in this disclosure may include an electroacoustic transducerconfigured to provide sounds from the at least one audio signal to anear of user 100. The electroacoustic transducer may include a speaker ora bone conduction microphone. In this example embodiment, transmissionmodule 4110 may transmit isolated audio signals to the hearing interfacedevice, and the conditioning of the audio signals may be performed bythe hearing interface device. In another example embodiment, the hearinginterface device may include a receiver configured to receive at leastone audio signal, wherein the at least one audio signal was acquired bya wearable microphone and was selectively conditioned by at least oneprocessor (e.g., processor 210 located in apparatus 110) configured toidentify an audioprint of the at least one sound-emanating object usinga plurality of reference audioprints, retrieve information from adatabase about the at least one sound-emanating object, and cause theconditioning based on retrieved information.

FIG. 41B is a schematic illustration showing an exemplary environment4150 for using a hearing aid system 4160 consistent with the presentdisclosure. Hearing aid system 4160 may include apparatus 110 andidentify one or more individuals within environment 4150 using wearablemicrophone 1720 and hearing interface device 1710 to provide selectivelyconditioned audio signals to an ear of user 100. In the illustratedscenario, apparatus 110 may identify a first individual 4152 and asecond individual 4154 using audioprints determined from recorded audiosignals. For example, wearable microphone 1720 may record audio signalsgenerated by sound-emitting objects in environment 4150. In someembodiments, the audio signals may represent voices of variousindividuals. For example, as shown in FIG. 41B, first audio signals 4156may represent a voice of first individual 4152 and second audio signals4158 may represent a voice of second individual 4154. The at least oneprocessing device of hearing aid system 4160 may analyze first audiosignals 4156 and second audio signals 4158 to separate them and todetermine audioprints associated with voices. For example, the at leastone processing device may use one or more speech or voice activitydetection (VAD) algorithms and/or the voice separation techniques toisolate audio signals associated with each voice. In some embodiments,the at least one processing device may perform further analysis on theaudio signal associated the detected voice activity to determine theaudioprint associated with each voice. For example, the at least oneprocessing device may use one or more voice recognition algorithms(e.g., Hidden Markov Models, Dynamic Time Warping, neural networks, orother techniques) to determine the audioprint associated with eachvoice. In some embodiments, as illustrated in FIG. 43B, the at least oneprocessing device may use captured images and one or more imagerecognition algorithms to identify an object, and thereafter theobject's identity may be used to determine the audioprint.

FIGS. 42A-42F are schematic illustrations of audio signals recordedduring the scenario illustrated in FIG. 41B and being processed by atleast one processing device using the software modules depicted in FIG.41A. In accordance with the present disclosure, audio analysis module4102 may receive audio signals acquired by wearable microphone 1720 thatreflect sounds generated by first individual 4152 and second individual4154. FIG. 42A illustrates audio stream 4200 acquired by wearablemicrophone 1720. Audio analysis module 4102 may also analyze audiostream 4200 to identify first audio signals 4156 associated with firstindividual 4152 and second audio signals 4158 associated with secondindividual 4154. FIG. 42B depicts first audio signal 4156 in light grayand second audio signal 4158 in dark gray. Audio analysis module 4102may further isolate first audio signal 4156 associated with firstindividual 4152 and second audio signal 4158 associated with secondindividual 4154. FIG. 42C depicts the two isolated audio signals. Afterthe at least one processing device determines the audioprints for audiosignals 4156 and 4158, identifies individuals 4152 and 4154, andretrieves from database 4112 information relating to individuals 4152and 4154, selective conditioning module 4108 may cause selectiveconditioning of first audio signal 4156 and second audio signal 4158. Inthe example illustrated in FIG. 42D, the retrieved information mayindicate that second individual 4154 is more important to user 100 thanfirst individual 4152. Accordingly, selective conditioning module 4108may attenuate first audio signal 4156 and amplify second audio signal4158. First conditioned audio signal 4202 was generated from first audiosignal 4156 and second conditioned audio signal 4204 was generated fromsecond audio signal 4158. Transmission module 4110 may receive firstconditioned audio signal 4202 and second conditioned audio signal 4204from selective conditioning module 4108 and may combine them together toa conditioned audio stream 4206, as illustrated in FIG. 42E. Thereafter,transmission module 4110 may cause transmission of conditioned audiostream 4206 to hearing interface device 1710 configured to providesounds to an ear of user 100. FIG. 42F depicts conditioned audio stream4206 as received by hearing interface device 1710. Consistent with thepresent disclosure, the at least one processing device may causetransmission of conditioned audio stream 4206 to hearing interfacedevice 1710 in less than 100 mSec after audio stream 4200 was acquiredby the wearable microphone. For example, conditioned audio stream 4206may be transmitted to the hearing interface device in less than 50 mSec,less than 30 mSec, less than 20 mSec, or less than 10 mSec after audiostream 4200 was acquired by the wearable microphone.

FIG. 43A is a flowchart showing an exemplary process 4300 forselectively conditioning audio signals associated with a recognizedobject consistent with the disclosed embodiments. Process 4300 may beperformed by one or more processors associated with apparatus 110, suchas processor 210. In some embodiments, some or all of process 4300 maybe performed by devices external to apparatus 110, such as hearinginterface device 1710 or computing device 120. In other words, theprocessing device performing process 4300 may include at least oneprocessor located in a single housing including the wearable camera andthe wearable microphone, or a plurality of processors located inseparate housings.

In step 4302, the processing device may receive audio signals acquiredby a wearable microphone. The audio signals may be representative ofsounds emanating from objects in an environment of user 100. Consistentwith the present disclosure, the received audio signals may include anyform of data generated in response to sounds within a range of between10 to 30,000 hertz (e.g., between 20 to 20,000 hertz) in the environmentof user 100. For example, the audio signals may represent soundsgenerated by multiple sound-emanating objects. Consistent with thepresent disclosure, the wearable microphone may include one or moredirectional microphones, a microphone array, a multi-port microphone, orvarious other types of microphones. The processing device may beconfigured to determine a directionality of sounds in the environment ofuser 100. Accordingly, the audio signals may be indicative of a regionof the environment of user 100 associated with a sound-emanating objectthat generated the sounds represented by the audio signals.

In step 4304, the processing device may analyze the received audiosignals to obtain an isolated audio stream associated with asound-emanating object in the environment of user 100. In oneembodiment, the processing device may analyze the received audio signalsby using audio sample convolution. Specifically, speaker separation andother audio analysis algorithms described in this disclosure may useaudio sample convolution. For example, by convoluting past samples whencalculating a value for a present sample, and avoiding waiting forfuture samples, the delay providing the analysis results may besignificantly reduced. For example, the delay in generating an isolatedaudio stream (or any other processed audio stream) for eachsound-emitting object may be less than 50 mSec (e.g., less than 10 mSec,less than 5 mSec, or less than 1 mSec). In the scenario illustrated inFIG. 41B, the processing device may generate an isolated audio streamfor each of the two speakers in front of user 100. Each isolated audiostream may include of the voice of a speaker isolated from any othersounds such as background noises or other voices.

In step 4306, the processing device may determine an audioprint from theisolated audio stream. In one embodiment, the determined audioprint maybe a voiceprint associated with an individual. In another embodiment,the determined audioprint may be associated with a non-humansound-emitting object, such as, AC, a car, an animal, etc. Thedetermination of the audioprint may be performed by extracting spectralfeatures, also referred to as spectral attributes, spectral envelope, orspectrogram from the isolated audio stream. In one embodiment, theisolated audio stream may be inputted into a computer-based model suchas a pre-trained neural network, which outputs audioprint based on theextracted features. The determined audioprint may be used to identifythe sound-emitting object to cause selective conditioning of itsassociated audio signals. Consistent with the present disclosure, theprocessing device may access at least one identification databasestoring a set of reference audioprints for different sound-emanatingobjects. The set of reference audioprints may be previously determinedby the processing device, or determined by a different processingdevice. The set of reference audioprints may be used in determining theaudioprint of the sound-emitting object. For example, the processingdevice may select the most similar audioprint from the set of referenceaudioprints as the determined audioprint of the sound-emitting object.In another embodiment, the set of reference audioprints may be used inidentifying the sound-emanating object. For example, the processingdevice may trigger a comparison between the determined audioprint andthe set of reference audioprints to determine an identity of thesound-emanating object.

In some cases, when the determined audioprint matches one of the sets ofreference audioprints, the processing device may cause the conditioningof at least one audio signal based on predefined settings associatedwith the identity of the sound-emanating object. In other cases, whenthe determined audioprint fails to match any of the sets of referenceaudioprints, the processing device may determine at least one indicatorof a level of similarity between a specific reference audioprint and thedetermined audioprint. Based on a comparison of the at least oneindicator of a level of similarity with a predetermined threshold, theprocessing device may cause the conditioning of at least one audiosignal based on predefined settings associated with the specificreference audioprint. In addition, when the determined audioprint failsto match any of the sets of reference audioprints, the processing devicemay determine at least one indicator of a level of similarity between aspecific reference audioprint and the determined audioprint. Based on acomparison of the at least one indicator of a level of similarity with apredetermined threshold, the processing device may update the set ofreference audioprints based on the determined audioprint. In someembodiments, the set of reference audioprints may include a plurality ofreference audioprints associated with a single sound-emitting object.For example, the set of reference audioprints may include a firstreference audioprint for a specific individual determined based on aninstance where the specific individual was standing next to user 100,and a second reference audioprint for the specific individual determinedbased on an instance where a voice of the specific individual wasprojected from a communication device.

In step 4308, the processing device may use the audioprint to retrievefrom a database information relating to the sound-emanating object. Inone embodiment, the retrieved information may be indicative of apre-existing relationship between user 100 and the sound-emanatingobject. Accordingly, the processing device cause selective conditioningof at least one audio signal based on the pre-existing relationship. Forexample, the processing device may apply a hierarchy of amplificationsto audio signals associated with a plurality of sound-emanating objectshaving various levels of pre-existing relationships. Consistent with thepresent disclosure, the retrieved information may include at least onepredefined audio conditioning parameter value to apply to the audiostream associated with the sound-emanating object. The at least onepredefined audio conditioning parameter may include pitch, loudness,cadence, smoothness, intonation, and more. In one embodiment, the atleast one predefined audio conditioning parameter value included in theretrieved information may be dependent on an audio hierarchy and thesound-emanating object's position in the audio hierarchy (e.g., a firealarm may be ranked higher than office chatter). In a first example, thevalue of the retrieved at least one predefined audio conditioningparameter may cause amplification of audio signals associated with aparticular sound-emanating object at a level higher than for anothersound-emanating object lower on the audio hierarchy than the particularsound-emanating object. In a second example, the value of the retrievedat least one predefined audio conditioning parameter may causeattenuation of the audio signals associated with a particularsound-emanating object at a level lower than for another sound-emanatingobject higher on the audio hierarchy than the particular sound-emanatingobject. In a third example, the value of the retrieved at least onepredefined audio conditioning parameter may cause a change in toneassociated with the audio signals associated with a particularsound-emanating object. In this example, objects higher in the hierarchymay receive tone modification while lower ranked objects may receive notone change.

In step 4310, the processing device may cause selective conditioning ofat least one audio signal received by the wearable microphone from aregion associated with the at least one sound-emanating object.Consistent with the present disclosure, the processing device maydetermine a type associated with the sound-emanating object based on acomparison of the determined audioprint with a set of referenceaudioprints. The type associated with the sound-emanating object mayinclude mechanical machines, speakers, humans, animals, inanimateobjects, weather-related objects, and more. After determining the typeassociated with the sound-emanating object, the processing device maycause the selective conditioning of the at least one audio signal basedon the determined type. In addition, the processing device may analyzethe received audio signals to isolate audio packets determined to beassociated with multiple sound-emanating objects in the environment ofuser 100. For example, the multiple sound-emanating objects may includea first individual and a second individual. Accordingly, the processingdevice may cause a first selective conditioning of audio signalsassociated with the first individual based on retrieved informationassociated with the first individual, and cause a second selectiveconditioning, different from the first selective conditioning of audiosignals, associated with the second individual based on retrievedinformation associated with the second individual. For example,amplifying audio signals associated with the first individual and pitchenhancement to audio signals associated with the second individual. FIG.42D shows how the processing device may cause a first selectiveconditioning to audio signals associated with the first individual and asecond selective conditioning to audio signals associated with thesecond individual.

In step 4312, the processing device may cause transmission of the atleast one conditioned audio signal to a hearing interface deviceconfigured to provide sounds to an ear of user 100. Consistent with thepresent disclosure, the processing device may cause a transmitter (e.g.,wireless transceiver 530 a) to transmit the conditioned audio signals tothe hearing interface device via a wireless network (e.g., cellular,Wi-Fi, Bluetooth®, etc.), or via near-field capacitive coupling, othershort-range wireless techniques, or via a wired connection. In addition,the processing device may cause transmission of unprocessed audiosignals together with the conditioned audio signals to the hearinginterface device.

FIG. 43B is a flowchart showing another exemplary process 4350 forselectively conditioning audio signals associated with a recognizedobject consistent with the disclosed embodiments. Similar to process4300, process 4350 may be performed by one or more processors associatedwith apparatus 110 or by devices external to apparatus 110. In otherwords, the processing device performing process 4350 may include atleast one processor located in a single housing including the wearablecamera and the wearable microphone, or a plurality of processors locatedin separate housings.

In step 4352, the processing device may receive a plurality of imagesfrom an environment of user 100 captured by a wearable camera. Forexample, the suggested system may include a processor (e.g., processor210) configured to receive a plurality of images of the environment ofuser 100 captured by an image sensor (e.g., image sensor 220).Consistent with the present disclosure, the plurality of images mayinclude frames of a video stream captured by the wearable camera.

In step 4354, the processing device may process the plurality of imagesto detect a sound-emanating object in at least one of the plurality ofimages, and in step 4356, the processing device may identify thesound-emanating object using the at least one of the plurality ofimages. As used herein, the term “detecting a sound-emanating object”may broadly refer to determining an existence of the sound-emanatingobject. For example, the system may determine the existence of aplurality of distinct sound-emanating objects. By detecting theplurality of sound-emanating objects, the system may acquire differentdetails relative to the plurality of sound-emanating objects (e.g., howmany sound-emanating objects are present in the environment of user100), but it does not necessarily gain knowledge of the type of object.In contrast, the term “identifying a sound-emanating object” may referto determining a unique identifier associated with a specificsound-emanating object that allows the system to uniquely access recordsassociated with the sound-emanating object in a database (e.g., database4112). In some embodiments, the identification may at least in part bemade based on visual characteristics of the sound-emanating objectderived from images captured by the wearable camera. For example, thesound-emanating object may be an individual speaking with user 100 andthe visual characteristics of the sound-emanating object may includefacial features of the individual. The unique identifier may include anycombinations of numbers, letters, and symbols. Consistent with thepresent disclosure, the terms “determining a type of a sound-emanatingobject” may also be used interchangeably in this disclosure withreference to the term “identifying a sound-emanating object.”

In step 4358, the processing device may use the determined identity ofthe sound-emanating object to retrieve from a database informationrelating to the sound-emanating object. In one embodiment, the retrievedinformation may include a reference audioprint or voice print associatedwith the recognized sound-emanating object. In another embodiment, theretrieved information may be indicative of a pre-existing relationshipbetween user 100 and the recognized sound-emanating object. Accordingly,the processing device may cause selective conditioning of at least oneaudio signal based on the pre-existing relationship. Additional detailson how the retrieved information may be used to cause selectiveconditioning of at least one audio signal are described above withreference to step 4308.

In step 4360, the processing device may receive at least one audiosignal acquired by a wearable microphone, wherein the at least one audiosignal is representative of sounds emanating from the sound-emanatingobjects. Consistent with some embodiments of the present disclosure, theidentities of one or more of the sound-emanating objects may bedetermined based on the received images from the wearable camera and theat least one audio signal acquired by the wearable microphone. Forexample, the at least one audio signal may be used together with thereceived imaged to identify one or more sound-emanating objects when aconfidence score corresponding to a degree of certainty that a soundemanating object represented in the captured images corresponds to oneor more objects in database is below a certain threshold.

In step 4361, the processing device may use retrieved information (e.g.,the information retrieved in step 4358) to process at least one audiosignal (e.g., the at least one audio signal received in step 4360). Inone embodiment, when the sound-emanating object is an individual, theretrieved information may include at least one detail about theindividual (e.g., gender, age, ethnicity, etc.). The processing devicemay use the at least one detail about the individual to separate soundsassociated with the individual from sounds emanating from other soundemanating objects. In another embodiment, the retrieved information mayinclude a reference audioprint associated with the recognizedsound-emanating object and the processing device may use the referenceaudioprint to identify and separate sounds associated with therecognized sound emanating object from sounds emanating from other soundemanating objects. Consistent with the present disclosure, audioseparation may be more efficient when the retrieved information includesa reference audioprint of a recognized individual, but for someimplementations of the system at least one detail about the individualmay be sufficient.

In step 4362, the processing device may cause selective conditioning ofthe audio signal received by the wearable microphone from a regionassociated with the at least one sound-emanating object as separated instep 4361. Thus, in this example, only the audio emanating from aparticular object, for example a person the user is speaking with, maybe conditioned. For example, the audio may be amplified.

In step 4364 the processing device may cause transmission of the atleast one conditioned audio signal to a hearing interface deviceconfigured to provide sounds to an ear of user 100. The detailsdescribed above with reference to steps 4310 and 4312 are relevant alsofor steps 4362 and 4364.

Selective Modification of Background Noises

Users of hearing aids systems typically find it intrusive whenirrelevant background noises are amplified. Some existing hearing aidssystems filter out low-frequency sounds to reduce background noises.This solution eliminates some of the background noises, but it providesa partial solution as it may eliminate important parts of speech soundsor other sounds in the environment of user 100. Other existing hearingaids systems use directional microphones to reduce the sounds frombeside and behind the user. This solution provides a bettersignal-to-noise ratio in certain specific scenarios, but it alsoprovides a partial solution, as some background noises are important andshould not be eliminated. The disclosed hearing aid system may include awearable device (e.g., apparatus 110) that causes selective conditioningof audio signals generated by a sound-emanating object in theenvironment of the user, and a hearing interface device (e.g., hearinginterface device 1710) to provide selectively modified sounds to an earof user 100. The disclosed hearing aid system may use image data todetermine if the background noises are important and cause selectiveconditioning accordingly. For example, the hearing aid system mayamplify background noises determined to be important and attenuatebackground noises determined not to be important.

FIG. 44A illustrates a scenario where user 100 is working at his desk.User 100 wears a hearing aid system 4400 that may include wearablecamera 4402, wearable microphone 4404, and hearing interface device4406. In the illustrated scenario, a first part of the environment ofuser 100 is associated with field of view 4408 of wearable camera 4402and may include at least one sound-emanating object, and a second partof the environment of user 100 may also include at least onesound-emanating object outside field of view 4408 of wearable camera4402. For example, the first part of the environment of user 100 mayinclude a first sound-emanating object 4410A (e.g., a computer withspeakers), and the second part of the environment of user 100 mayinclude a second sound-emanating object 4412A (e.g., a woman) and athird sound-emanating object 4414A (e.g., a television).

FIG. 44B illustrates audio signals 4416 acquired by wearable microphone4404 during a time period T. As shown in the figure, acquired audiosignals 4416 include first audio signals 4410B from firstsound-emanating object 4410A, second audio signals 4412B from secondsound-emanating object 4412A, and third audio signals 4414B from thirdsound-emanating object 4414A. In the scenario described above, hearingaid system 4400 may determine that the sounds from secondsound-emanating object 4412A are more important that the sounds fromthird sound-emanating object 4414A, and attenuate third audio signals4414B generated by third sound-emanating object 4414A. FIG. 44Cillustrates conditioned audio signals 4418 transmitted to hearinginterface device 4406. Conditioned audio signals 4418 includes firstaudio signals 4410C, second audio signals 4412C, and third audio signals4414C. In the illustrated example, only the third audio signals 4414Care conditioned; specifically, third audio signals 4414C were attenuatedbecause hearing aid system 4400 determined that the sounds from secondsound-emanating object 4412A are more important that the sounds fromthird sound-emanating object 4414A.

In one embodiment, hearing aid system 4400 may use image data capturedby wearable camera 4402 during time period T to determine the importanceof the audio signals. For example, hearing aid system 4400 may determinefrom the image data that user 100 is sitting in his office and use thisinformation to identify the woman based on her voice as his supervisor.Hearing aid system 4400 may determine the importance of the sounds fromthe woman based on her identity. In another embodiment, the image datamay be captured by wearable camera 4402 before time period T. Forexample, while user 100 walked to his desk, or sat at his desk andturned around, wearable camera 4402 captured at least one image of thewoman participating in an activity. Hearing aid system 4400 maydetermine the importance of the sounds from the woman based on theactivity the woman participated in.

FIG. 45 is a block diagram illustrating the components of hearinginterface device 4406 configured to communicate with apparatus 110 andcomputing device 120, according to example embodiments. As shown in FIG.45, hearing interface device 4406 may include a receiver 4500, anelectroacoustic transducer 4502, a processor 4504, a memory 4506, and amobile power source 4508. Receiver 4500 may be used to receive data(e.g., audio signals, data about sound-emitting objects, and more) fromapparatus 110 and/or from computing device 120. Electroacoustictransducer 4502 may be used to generate sounds based on the receiveddata. The generated sounds may be provided to an ear of user 100. In oneembodiment, electroacoustic transducer 4502 may include a speaker. Inanother embodiment, electroacoustic transducer 4502 may include a boneconduction microphone. Processor 4504, memory 4506, and mobile powersource 4508 may operate in a manner similar to processor 210, memory550, and mobile power source 520 described above. As will be appreciatedby a person skilled in the art, having the benefit of this disclosure,numerous variations and/or modifications may be made to hearinginterface device 4406. Not all of the components included in theillustrated configuration of hearing interface device 4406 are essentialfor the operation of hearing aid system 4400. Any component may belocated in any appropriate apparatus and the components may berearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, in oneconfiguration, hearing interface device 4406 may include a processor forselective conditioning of received audio signals. In anotherconfiguration, hearing interface device 4406 may receive audio signalsselectively conditioned by a processor located in a separate device(e.g., apparatus 110 or computing device 120).

In one embodiment, receiver 4500 may receive at least one audio signal.The at least one audio signal may have been acquired by a wearablemicrophone (e.g., wearable microphone 4404). The at least one audiosignal may have been selectively conditioned by at least one processor(e.g., processor 210 or processor 540). The at least one processor mayreceive a plurality of images captured by a wearable camera (e.g.,wearable camera 4402) and determine, based on an analysis of theplurality of images, that the at least one sound was generated by aremote sound-emanating object outside of a field of view of the wearablecamera. Thereafter, the at least one processor may cause theconditioning of the audio signal based on information about the remotesound-emanating object retrieved from at least one memory (e.g., memory550 a, memory 550 b, or memory 4506). Consistent with embodiments of thepresent disclosure, processor 4504 of hearing interface device 4406 mayhave at least some of the capabilities of: selective conditioning audiosignals, processing image data to identify objects, and processing audiodata to recognize sounds. Accordingly, the functionality described inthis disclosure with reference to a processing device located inapparatus 110 may be also executed by processor 4504. For example,receiver 4500 may receive nonconditioned audio signals acquired bywearable microphone 4404 and, thereafter, processor 4504 may determinethe importance of at least one audio signal and cause selectiveconditioning of the at least one audio signal based on informationretrieved from at least one memory.

FIG. 46A is an illustrative process 4600 for causing selectivemodification of background noises based on determined importance levels.The importance level assigned to audio signals associated with abackground noise may represent the likelihood that user 100 may beinterested in hearing said background noise. Consistent with the presentdisclosure, the importance levels assigned to audio signals associatedwith background noises may be determined based on the content of thebackground noises, the identity of the sound-emitting objects thatgenerate the background noises, the context of the background noises,and more. For example, the importance level of a background noise may beclassified into multiple levels, such as “nuisance,” “relevant,” and“critical.” Alternatively, the importance level may be represented as anumeric value between one and ten, where one is not important at all andten is very important.

At step 4602, the hearing aid system (e.g., hearing aid system 4400) mayreceive audio signals including background noises. The background noisesmay include sounds from one or more sound-emanating objects in theenvironment of user 100, but outside of field of view 4408, for example,sounds from second sound-emanating object 4412A and sounds from thirdsound-emanating object 4414A. The audio signals may be receivedseparately, for example through directional microphones. Additionally,or alternatively, the audio signals may be received together andthereafter separated, using for example known audio prints of a specifichuman, known patterns, for example the sound on an A/C, or the like.

At step 4604, the hearing aid system may identify one or moresound-emanating objects responsible for at least some of the backgroundnoises. Consistent with the present disclosure, the hearing aid systemmay identify the sound-emanating object using information from one ormore images captured by wearable camera 4402. For example, the hearingaid system may determine from captured image data that user 100 is athis home and use this information to determine the identity of thesound-emanating object responsible for at least some of the backgroundnoises.

At step 4606, the hearing aid system may determine an importance levelassociated with the sounds from the sound-emanating object. In oneembodiment, the determination of the importance level may be based on adetected voiceprint of the sound-emanating object. In anotherembodiment, the determination of the importance level may be based on ananalysis of a plurality of images captured by wearable camera 4402. Forexample, when wearable microphone 4404 detects audio signals associatedwith a car honk when user 100 walks down the street, the hearing aidsystem may rank the importance level to these audio signals as 7.5.Alternatively, when wearable microphone 4404 detects audio signalsassociated with a car honk when user 100 sits in a restaurant, thehearing aid system may rank the importance level to these audio signalsas 2.8. In addition, the determination of the importance level may bebased on context derived from analysis of at least one image captured bywearable camera 4402 before receiving the background noises. Forexample, the hearing aid system may determine that user 100 may put ababy in bed based on image analysis and, five minutes later, wearablemicrophone 4404 may detect a baby crying. In this case, the hearing aidsystem may categorize the audio signals associated with the baby cryingas “critical.” In another embodiment, the determination of theimportance level may be based on context derived from audio signalsrepresentative of sounds acquired before receiving the sounds from thesound-emanating object. For example, a person related to user 100 mayhave asked user 100 to take care of a baby and, five minutes later,wearable microphone 4404 may detect a baby crying. In this case, thehearing aid system may categorize the audio signals associated with thebaby crying as “critical.” In another case, the hearing aid system maydetermine user 100 is on a plane and that the baby crying in thebackground is not related to user 100. In this case, the hearing aidsystem may categorize the audio signals associated with the baby cryingas “nuisance.”

At step 4608, the hearing aid system may determine if the importancelevel is greater than a threshold based on retrieved information. Theterm “threshold” is used herewith to denote a reference value, a level,a point, or a range of values such that when the importance level isabove it the hearing aid system may follow a first course of action andwhen the importance level is under it the hearing aid system follows asecond course of action. The value of the threshold may be predeterminedfor each sound-emitting object or dynamically selected based on thecontext determined based on image data.

If the importance level is determined to be less than the threshold, thehearing aid system may cause a first selective conditioning to weakenaudio signals associated with the sound-emanating object (step 4610). Inone embodiment, when the hearing aid system determines that animportance level of the sound-emanating object is lower than thethreshold, the first selective conditioning may include attenuatingaudio signals associated with the sound-emanating object. For example,background noises from the AC may be considered to be unimportant, sothey can be silenced relative to sounds from other sound-emanatingobjects. Alternatively, the first selective conditioning may includeamplifying audio signals associated with other sound-emanating objects.Consistent with the present disclosure, the hearing aid system maydetermine that the importance level of the at least one sound is lowerthan the threshold based on analysis of a plurality of images fromwearable camera 4402.

If the importance level is determined to be greater than the threshold,the hearing aid system may cause a second selective conditioning tointensify audio signals associated with the sound-emanating object (step4612). In one embodiment, when the hearing aid system determines that animportance level of the sound-emanating object is greater than thethreshold, the second selective conditioning may include amplifyingaudio signals associated with the sound-emanating object. For example,background noises from certain colleagues may be considered to beimportant, so they can be amplified relative to sounds from othersound-emanating objects. Alternatively, the second selectiveconditioning may include attenuating audio signals associated with othersound-emanating objects.

In one embodiment, the hearing aid system may determine that thebackground noises were generated by a plurality of sound-emanatingobjects outside the field of view of the wearable camera. Consistentwith this embodiment, the hearing aid system may identify the pluralityof sound-emanating objects and rank the plurality of remotesound-emanating objects based on their corresponding importance levels.Thereafter, the hearing aid system may cause selective conditioning of aplurality of sounds associated with the plurality of remotesound-emanating objects based on their corresponding importance levels.For example, with reference to FIG. 44A, the hearing aid system may rankthe sounds from third sound-emanating object 4414A with a lowerimportance level than the sounds from third sound-emanating object 4414.Accordingly, the audio signals from second sound-emanating object 4412Amay be amplified and the audio signals from third sound-emanating object4414A may be attenuated.

After selective conditioning of audio signals associated with thesound-emanating object, the hearing aid system may provide theconditioned audio signals to user 100 (step 4614). The conditioned audiosignals may be provided to user 100 using electroacoustic transducer4502 of hearing interface device 4406. In one embodiment, the hearingaid system may notify user 100 about background noises that weresubstantially removed in the conditioned audio signals. For example, thehearing aid system may send to computing device 120 an indication aboutat least one sound-emitting object that its audio signals wereattenuated. After receiving a feedback from user 100 regarding the atleast one sound-emitting object, hearing aid system may avoidattenuating audio signals of the at least one sound-emitting object inthe future.

FIG. 46B is a flowchart showing an exemplary process 4650 for selectivemodification of different types of background noises consistent withdisclosed embodiments. Process 4650 may be performed by one or moreprocessors associated with apparatus 110, such as processor 210. In someembodiments, some or all of process 4650 may be performed by processorsexternal to apparatus 110, such as processor 4504 in hearing interfacedevice 4406 or processor 540 in computing device 120. In other words,the at least one processor performing process 4650 may be included inthe same common housing as wearable camera 4402 and wearable microphone4404 or may be included in a separate housing.

In step 4652, a processing device (e.g., processor 210) may receiveimage data from an environment of user 100 captured by wearable camera4402 during a time period. Consistent with the present disclosure, thereceived image data may include any form of data retrieved from opticalsignals in the near-infrared, infrared, visible, ultraviolet spectrumsor multi-spectral. The image data may include video clips, one or moreimages, or information derived from processing one or more images. Forexample, the image data may include details about objects (e.g.,sound-emanating objects and non-sound-emanating objects) identified inimages captured by wearable camera 4402.

In step 4654, the processing device may receive at least one audiosignal representative of sounds acquired by wearable microphone 4404during the time period. Consistent with the present disclosure, wearablemicrophone 4404 may include microphone array and/or at least onedirectional microphone for capturing sounds from at least onesound-emanating object in the environment of user 100. As used herein,the term “sound-emanating object” may refer to any object capable ofgenerating sounds within a range of between 10 to 30,000 hertz (e.g.,between 20 to 20,000 hertz). Examples of sound-emanating objects mayinclude different inanimate things (e.g., fans, speakers, traffic, wind,rain, etc.) and animate beings (e.g., people, animals). In oneembodiment, the at least one audio signal may include a plurality ofaudio signals from multiple sound-emanating objects, each audio signalhaving a distinct tone, distinct cadence, distinct loudness, or distinctcombination of tone, cadence, and loudness.

In step 4656, the processing device may determine that at least one ofthe sounds was generated by a sound-emanating object in the environmentof the user, but outside of a field of view of wearable camera 4402. Thesound-emanating object may be outside of the field of view of wearablecamera 4402 when it generates sounds and/or when the conditioned audiosignals are being transmitted to hearing interface device 4406. Theprocessing device may determine that at least one of the sounds wasgenerated by a sound-emanating object outside of the field of view ofwearable camera 4402 by identifying the objects in the field of view ofwearable camera 4402 and determining that the at least one of the soundswas not generated by any of the identified objects. The processingdevice may also determine that at least one of the sounds was generatedby a sound-emanating object outside of the field of view of wearablecamera 4402 using information about objects in the field of view ofwearable camera 4402 (e.g., voiceprint, relationship, and more)retrieved from a database, or objects that are not in the field of view,but have been identified earlier in the field of view.

Consistent with the present disclosure, the processing device mayanalyze the at least one audio signal to determine an importance levelof the sounds generated by a sound-emanating object outside of the fieldof view of wearable camera 4402. In one embodiment, the at least onesound may be associated with spoken words, and the processing device mayidentify at least one of the spoken words and determine an importancelevel of the at least one sound based on the identity of at least one ofthe spoken words. For example, the spoken words “help,” “be careful,”and the user's name may be associated with a higher importance levelthan other words. In another embodiment, the at least one soundgenerated by the sound-emanating object may be associated with afrequency range, and the processing device may determine an importancelevel of the at least one sound based on the detected frequency range.For example, a smoke alarm has a specific frequency and audio signalswith that specific frequency may be associated with a higher importancelevel than other audio signals. For example, the processing device maydetermine an importance level of a siren based on context, e.g., acertain siren may be more important when user 100 is walking in thestreet than when user 100 is indoors.

In step 4658, the processing device may retrieve from a databaseinformation associated with the at least one sound. The database may beany device capable of storing information about one or moresound-emanating objects, and may include a hard drive, a solid-statedrive, a web storage platform, a remote server, or the like. Thedatabase may be located within apparatus 110 (e.g., within memory 550 a)or external to apparatus 110 (e.g., within memory 550 b or within memory4506). In some embodiments, the database may be compiled by apparatus110 through previous audio analyses. For example, the processing devicemay store in the database information associated with voices and soundsrecognized in audio signals captured by wearable microphone 4404. Forexample, each time a voice detected in the audio signals is recognizedas complying with a stored voiceprint, the processing device storesinformation associated with the detected sound-emanating object, forexample an updated voice print. The processing device may retrieveinformation by analyzing the audio signals and identifying thevoiceprint of the sound-emanating object. The retrieved information mayinclude details associated with the identity of the sound-emittingobject. Specifically, in one embodiment, the retrieved information maybe indicative of a pre-existing relationship of user 100 with thesound-emanating object, and the at least one processor may be furtherprogrammed to determine an importance level of the at least one soundbased on the pre-existing relationship. For example, the woman askingfor help in FIG. 44A may be the user's supervisor. In anotherembodiment, the processing device may determine, based on analysis ofthe at least one of the sounds, that the at least one of the sounds isrelated to a public announcement. For example, the analysis of the atleast one of the sounds includes identifying a recognized word or phraseassociated with the public announcement. Moreover, the processing devicemay determine the relevancy of the public announcement to user 100 basedon automatic review of calendar data associated with user 100. Therelevancy of the public announcement to user 100 may affect thedetermination of the importance level. For example, the processingdevice can access calendar data to determine that the user is on flight641 to X destination on a certain day and time and selectively amplifyannouncements for this flight.

In step 4660, the processing device may cause selective conditioning ofaudio signals acquired by wearable microphone 4404 during the timeperiod based on the retrieved information. Consistent with the presentdisclosure, the conditioning may include amplifying audio signalsdetermined to correspond to the sound-emanating object outside of thefield of view of wearable camera 4402 relative to other audio signalsand/or optionally attenuation or suppression of one or more audiosignals associated with a sound-emanating object inside the field ofview of wearable camera 4402. Additionally, or alternatively, selectiveconditioning may include attenuation of audio signals determined tocorrespond to the sound-emanating object outside of the field of view ofwearable camera 4402 relative to other audio signals and/or optionallyamplifying one or more audio signals associated with a sound-emanatingobject inside the field of view of wearable camera 4402. Additionally,or alternatively, selective conditioning may include changing a tone orrate of speech associated with the audio signals determined tocorrespond to the sound-emanating object outside of the field of view ofwearable camera 4402 relative to other audio signals to make the soundmore perceptible to user 100 (e.g., increasing spaces between words,diction improvement, accent improvement, and more). Various otherprocessing may be performed such as digitally reducing noise within theaudio signal. Consistent with the present disclosure, the processingdevice may distinguish between three types of background noises. Forexample, the first type may be a stationary noise that is substantiallyconstant over time, such as a refrigerator. The second type may benonstationary noise that is relatively transient, such as the sound of afalling object. The third type may be temporary noise that is longer intime than the second type and shorter in time than the first type.Examples of the third type of background noise may include a passingcar, humming in an audience, and more. The processing device may causeselective conditioning of audio signals based on the identified type ofbackground noise.

As described above, the processing device may determine, based on theretrieved information, that an importance level of the at least onesound is greater than a threshold. In this embodiment, the selectiveconditioning of the audio signals may include amplifying the at leastone sound based on the determination of the importance level. Forexample, the retrieved information may identify some audio signals as afire alarm and rank these audio signals as important. When user 100 hasa lower sensitivity to tones in a range associated with the fire alarm,the selective conditioning of the audio signals may include changing atone of the audio signals to make the fire alarm more perceptible touser 100. In another embodiment, the selective conditioning furtherincludes attenuating sounds generated by other sound-emanating objects.The other sound-emanating objects may be inside or outside the field ofview of the camera.

In step 4662, the processing device may cause transmission of theconditioned audio signals to hearing interface device 4406, which may beconfigured to provide sounds to an ear of user 100. Consistent with thepresent disclosure, the processing device may cause a transmitter (e.g.,wireless transceiver 530 a) to transmit the conditioned audio signals tohearing interface device 4406 via a wireless network (e.g., cellular,Wi-Fi, Bluetooth®, etc.), or via near-field capacitive coupling, othershort-range wireless techniques, or via a wired connection. In addition,the processing device may cause transmission of unprocessed audiosignals together with the conditioned audio signals to hearing interfacedevice 4406. In one embodiment, the conditioned audio signals may betransmitted to hearing interface device 4406 in less than 100 mSec afterthe at least one audio signal was acquired by wearable microphone 4404.For example, the conditioned audio signals may be transmitted to hearinginterface device 4406 in less than 50 mSec, less than 30 mSec, less than20 mSec, or less than 10 mSec after the at least one audio signal wasacquired by wearable microphone 4404.

Using Voice and Visual Signatures to Identify Objects

Consistent with the disclosed embodiments, a hearing aid system may usevoice and visual signatures to identify objects within an environment ofa user. The hearing aid system may analyze captured images of theenvironment of a user to identify a sound-emanating object and determinevisual characteristics of the object. When the identification is notcertain (e.g., a confidence level is below a predetermined level), orbased on any other criteria, the system may use a voiceprint determinedfrom acquired audio signals to identify the object. The hearing aidsystem may repeat one or more portions of this process until a certaintyexceeds a threshold, and then take an action based on the determinedidentity of the object. This association between visual and audioidentification provides for faster start of audio analysis actions suchas speaker separation.

FIG. 47A is a block diagram illustrating a hearing aid system 4700according to an example embodiment. Hearing aid system 4700 may includeat least one wearable camera 4701, at least one microphone 4702, atleast one processor 4703, and at least one memory 4704. Hearing aidsystem 4700 may further include additional components beyond those shownin FIG. 47A. For example, hearing aid system 4700 may include one ormore components described above with respect to FIGS. 5A-5C. Further,the components shown in FIG. 47A may be housed in a single device or maybe contained in one or more different devices.

Wearable camera 4701 may be configured to capture one or more imagesfrom the environment of user 100. In some embodiments, wearable camera4701 may be included in a wearable camera device, such as apparatus 110.For example, wearable camera 4701 may be camera 1730, as describedabove, which may also correspond to image sensor 220.

Microphone 4702 may be configured to capture sounds from the environmentof user 100. In some embodiments, camera 4701 and microphone 4702 may beincluded in the same device. Similar to wearable camera 4701, microphone4702 may be included in a wearable camera device, such as apparatus 110.For example, apparatus 110 may comprise microphone 1720, as describedwith respect to FIG. 17B, which may be configured to determine adirectionality of sounds in the environment of user 100. As discussedabove, apparatus 110 may be worn by user 100 in various configurations,including being physically connected to a shirt, necklace, a belt,glasses, a wrist strap, a button, or other articles associated with user100. In some embodiments, one or more additional devices may also beincluded, such as computing device 120. Accordingly, one or more of theprocesses or functions described herein with respect to apparatus 110 orprocessor 210 may be performed by computing device 120 and/or processor540. Apparatus 110 may also communicate with a hearing interface deviceworn by user 100, such as hearing interface device 1710. Suchcommunication may be through a wired connection, or may be madewirelessly (e.g., using a Bluetooth™, NFC, or forms of wirelesscommunication).

Processor 4703 may be configured to receive and process images and audiosignals captured by wearable camera 4701 and microphone 4702. In someembodiments, processor 3803 may be associated with apparatus 110, andthus may be included in the same housing as wearable camera 4701 andmicrophone 4702. For example, processor 4703 may correspond toprocessors 210, 210 a or 210 b, as described above with respect to FIGS.5A and 5B. In other embodiments, processor 4703 may be included in oneor more other devices, such as computing device 120, server 250 (FIG. 2)or various other devices. In such embodiments, processor 4703 may beconfigured to receive data remotely, such as images captured by wearablecamera 4701 and audio signals captured by microphone 4702.

Memory 4704 may be configured to store information associated with soundemanating objects in the environment of user 100. Memory 4704 may be anydevice capable of storing information about one or more objects, and mayinclude a hard drive, a solid state drive, a web storage platform, aremote server, or the like. Memory 4704 may be located within apparatus110 (e.g., within memory 550) or external to apparatus 110.

FIG. 47B is a schematic illustration showing an exemplary environmentfor using voice and visual signatures to identify objects consistentwith the present disclosure. The environment of user 100 may include oneor more sound-emanating objects. The sound emanating objects may includeany objects capable of emitting sounds that are perceptible to user 100or apparatus 110. For example, the sound emanating objects may be soundemanating objects 4710 and 4711, shown in FIG. 47. In some instances,sound emanating objects 4710 or 4711 may be an individual, as shown inFIG. 47. In other embodiments sound emanating objects 4710 or 4711 maybe a device, such as a radio, a speaker, a television, a mobile device(e.g., a mobile phone, tablet, etc.), a computing device (e.g., personalcomputer, desktop computer, laptop, gaming console, etc.), vehicles,alarms, or any other device capable of emitting sounds. Thesound-emanating objects 4710 or 4711 may also include other objects,such as pets, animals, insects, natural features (e.g., streams, trees,etc.) or any other objects that may emanate sounds.

Hearing aid system 4700 may be configured to receive images and/or audiosignals associated with sound emanating objects 4710 and/or 4711. Forexample, wearable camera 4701 may be included in apparatus 110, worn byuser 100. Wearable camera 4701 may capture an image including arepresentation of sound emanating object 4710 within the environment ofuser 100. The image may contain representations of other objects orfeatures within the environment of user 100. Processor 4703 may receivea plurality of images captured by wearable camera 4701 and analyze theimages to determine visual characteristics of sound emanating object4710. Such visual characteristics may include any features of the objectrepresented in the image. For example, the visual characteristics mayinclude a color, shape, size, or the like. In some embodiments thevisual characteristics may be indicative of a type of the soundemanating object. For example, the visual characteristics may identifywhether sound emanating object 4710 is an individual or an inanimateobject, a classification of the object (e.g., television, vehicle,animal, person, etc.), an identity of an individual, an identity of theobject, or other similar object type classifications. Accordingly,processor 4703 may use one or more image recognition techniques oralgorithms to detect features of sound emanating object 4710. Forexample, processor 4703 may identify one or more points, edges, verticesor other features of the object. For example, where sound emanatingobject 4710 is an individual, processor 4703 may further determine thevisual characteristics based on a facial analysis of an image of theindividual. Accordingly, processor 4703 may identify facial features onthe face of the individual, such as the eyes, nose, cheekbones, jaw, orother features. Processor 4703 may use one or more algorithms foranalyzing the detected features, such as principal component analysis(e.g., using eigenfaces), linear discriminant analysis, elastic bunchgraph matching (e.g., using Fisherface), Local Binary PatternsHistograms (LBPH), Scale Invariant Feature Transform (SIFT), Speed UpRobust Features (SURF), or the like. Similar feature recognitiontechniques may be used for detecting features of inanimate objects aswell.

Processor 4703 may further be configured to receive audio signalsassociated with sound emanating objects in the environment of user 100.The audio signals may be representative of one or more sounds emanatingfrom the sound emanating object. For example, sound emanating object4710 may emanate sound 4720, as shown in FIG. 47B. Microphone 4702 maybe configured to capture sound 4720 and convert it to an audio signal tobe processed by processor 4703. Sound 4720 may be any sound or noiseproduced by sound emanating object 4710. For example, sound 4720 may bean output of a television, mobile phone, or other device, or a soundproduced by a vehicle. In instances where sound emanating object 4710 isan individual, sound 4720 may be a voice of the individual. Processor4703 may be configured to analyze the received audio signals todetermine a voiceprint of the sound emanating object. Processor 4703 maybe configured to determine the voiceprint based on audio analysis of arecording of the individual. This may be performed using a voicerecognition component, such as voice recognition component 2041, asdescribed in FIG. 20B. Processor 4703 may use one or more voicerecognition algorithms (e.g., Hidden Markov Models, Dynamic TimeWarping, neural networks, or other techniques) to recognize the voice ofthe individual. The determined voiceprint may include variouscharacteristics associated with the individual, such as an accent of theindividual, the age of the individual, a gender of the individual, orthe like. While the voiceprint may represent a voice pattern of anindividual, the term voiceprint should be interpreted broadly to includeany sound pattern or feature that may be used to identify soundemanating object 4710.

Memory 4704 may include one or more databases 4705 containing referencevisual characteristics and reference voiceprints corresponding to aplurality of objects. For example, database 4705 may store a pluralityof visual characteristics and may associate one or more objects with thevisual characteristics. For example, database 4705 may associate a size,color, shape, or other visual characteristics with a particular type ofobject, such as a television or mobile phone. Database 4705 may alsoassociate visual characteristics with a specific object, rather than anobject type. For example, visual characteristics may be used to identifya mobile phone or other object as belonging to user 100 or anotherindividual known to user 100. In some embodiments, database 4705 mayinclude a list of contacts known to user 100. Visual characteristics mayinclude facial features used to identify a particular individual. Insome embodiments, database 4705 may be associated with a social networkplatform, such as Facebook™, LinkedIn™, Instagram™, etc. Processor 4703may be configured to access database 4705 to identify sound emanatingobject 4710. For example, processor 4703 may compare visualcharacteristics determined from the captured images to visualcharacteristics stored within database 4705. Processor 4703 maydetermine a match between the sound emanating object represented in theimages and an object in database 4705 based on how closely the visualcharacteristics match. In some embodiments, processor 4703 may furtherbe configured to determine a confidence score associated with the match.For example, the confidence score may be based on the number of visualcharacteristics detected in the image that match visual characteristicsin the database for a given object. The confidence score may also bebased on the degree to which the visual characteristics match those indatabase 4705. For example, if the visual characteristic is a color, theconfidence score may be based on how closely a color detected in theimage matches a color represented in database 4705. The confidence scoremay be represented on a scale (e.g., ranging from 1-10, 1-100, etc.), asa percentage or any other suitable format. In some embodiments,identifying the object may comprise comparing the confidence score to acertain threshold value, or determining a confidence score for multiplepotential objects and selecting the object with the highest score.

Database 4705 may similarly associate voiceprint data with a pluralityof objects. For example, database 4705 may contain voiceprint dataassociated with a number of individuals, similar to the stored visualcharacteristic data described above. For example, processor 4703 maycompare voiceprint data determined from the received audio signals tovoiceprint data within database 4705. Processor 4703 may determine amatch between the sound emanating object represented in the audiosignals and an object in database 4705 based on how closely thevoiceprint data matches. This process may be used alone, or inconjunction with the visual characteristic identification techniquesdescribed above. For example, sound emanating object may be recognizedusing the visual characteristics and may be confirmed using thevoiceprint data, or vice versa. In some embodiments the identificationof the at least one sound emanating object using the determined visualcharacteristics may result in a group of candidate objects, and theidentification of the at least one sound emanating object may includeselecting one candidate of the group of candidate objects based on thevoiceprint.

Similar to the visual characteristics, processor 4703 may further beconfigured to determine a confidence score associated with thevoiceprint match. For example, the confidence score may be based on thedegree to which the voiceprint detected in the audio signals matchesvoiceprint data stored in database 4705 for a given object. In someembodiments, the confidence score for the voiceprint data may becombined with the confidence score based on the visual characteristics,described above. For example, a single confidence score may representthe degree of confidence that sound emanating object 4710 correspondswith an object in database 4705 based on combined analysis of the visualcharacteristics and the voiceprint. In some embodiments, processor 4703may determine a confidence score based on the visual characteristicsand, if the confidence score does not exceed a certain threshold, usethe voiceprint data to further identify sound emanating object 4710 andrefine the confidence score.

Consistent with the present disclosure, database 4703 may be built atleast in part through a machine learning process. For example, database4703 may be compiled by inputting a training data set into a trainingalgorithm to associate various visual characteristics or voiceprintswith known objects. Accordingly, identifying the sound emanating objectmay be based on an output of a trained neural network associated withdatabase 4705. The trained neural network may be continuously improvedas hearing aid system 4700 continues to identify objects. For example,user 100 may confirm or manually edit the identity of objects identifiedby processor 4703 and the neural network may be adjusted or furtherdeveloped based on the feedback from user 100. Such feedback may bereceived through a device associated with user 100, such as apparatus110, computing device 120, or any other device capable of interactingwith hearing aid system 4700 (e.g., through a network connection, etc.).

In some embodiments, processor 4703 may be configured to determine theidentity of a sound emanating object based on visual characteristics orvoiceprint data associated with another sound emanating object. Forexample, the at least one sound emanating object may include a firstsound emanating object (e.g., sound emanating object 4710) and a secondsound emanating object (e.g., sound emanating object 4711). Hearing aidsystem 4700 may use determined visual characteristics of the first soundemanating object to identify the second sound emanating object.Similarly, hearing aid system 4700 may use the determined voiceprint ofthe first sound emanating object to identify the second sound emanatingobject. The visual characteristics or voiceprint data from first soundemanating object 4710 may be indicative of the identity of second soundemanating object 4711. For example, where the sound emanating objectsare individuals, a first individual may frequently be encountered alongwith the second individual. As another example, an individual mayfrequently be associated with an object such as a mobile phone, pet, orthe like. Processor 4703 may determine the identity of the object basedon visual characteristics (e.g., face recognition) and voiceprint dataassociated with the individual. Accordingly, database 4705 (or memory4704) may be configured to store associations between various objectswithin the database.

Processor 4703 may be configured to adjust the confidence score based onwhether second sound emanating object 4711 was identified based onvisual characteristics and/or voiceprint data of first sound emanatingobject 4710. For example, where second emanating object 4711 wasidentified based on visual characteristics and/or voiceprint data offirst sound emanating object 4710 alone, processor 4703 may assign alower confidence score. One the other hand, where second sound emanatingobject 4711 was identified based on visual characteristics and/orvoiceprint data associated with second sound emanating object 4711 andconfirmed using visual characteristics and/or voiceprint data associatedwith first sound emanating object 4710, processor 4703 may assign ahigher confidence score than if second sound emanating object 4711 wasidentified based on its own visual characteristics and/or voiceprintalone.

In some embodiments, hearing aid system 4700 may be configured toperform various actions based on identifying sound emanating object4710. In some embodiments, processor 4703 may store information relatingto the identification of 4710. For example, processor 4703 may store inmemory 4704 information relating to an encounter with sound emanatingobject 4710. This may include storing information such as the identityof the object (or identity of an individual) determined above. Theinformation may further include a time associated with theidentification, a time associated with the image or audio signal beingcaptured, a location (e.g., of user 100 or of sound emanating object4710), data associated with the sound emanating object (e.g., thecaptured images or audio signals, etc.), keywords mentioned in anencounter, or various other information. In some embodiments, processor4703 may maintain a timeline of identified objects or other eventsassociated with apparatus 110, and processor 4703 may add the identifiedsound emanating object to the timeline. In some embodiments, storing theinformation may include updating database 4705. For example, theinformation may be used for updating the visual characteristics of soundemanating object 4710 or may be used for updating the voiceprint ofsound emanating object 4710. The stored information may improve theaccuracy of the associations stored in database 4705 and thereby improvethe accuracy of hearing aid system 4700 in future object identification.

In some embodiments, hearing aid system 4700 may be configured tocondition the sound received from the sound emanating object. In someembodiments, the action performed by hearing aid system 4700 may includecausing selective conditioning of at least one audio signal associatedwith the at least one sound emanating object and causing transmission ofthe at least one conditioned audio signal to a hearing interface deviceconfigured to provide sounds to an ear of the user. For example,processor 4703 may receive an audio signal associated with sound 4720from sound emanating object 4710. Based on the identification of soundemanating object 4710, processor 4703 may selectively condition theaudio signal associated with sound 4720. For example, sound emanatingobject 4710 may be a television and processor 4703 may selectivelycondition the audio of the television. Where sound emanating object 4710is an individual, processor 4703 may determine that sound 4720 emanatingfrom the individual should be selectively conditioned.

In some embodiments, conditioning may include changing a tone of one ormore audio signals corresponding to sound 4720 to make the sound moreperceptible to user 100. For example, user 100 may have lessersensitivity to tones in a certain range and conditioning of the audiosignals may adjust the pitch of sound 4720. For example, user 100 mayexperience hearing loss in frequencies above 10 kHz and processor 210may remap higher frequencies (e.g., at 15 kHz) to 10 kHz. In someembodiments processor 210 may be configured to change a rate of speechassociated with one or more audio signals. Processor 210 may beconfigured to vary the rate of speech of sound emanating object 4710 tomake the detected speech more perceptible to user 100. The type anddegree of selective conditioning may depend on the particular object orindividual that was identified and/or on preferences of the user. Forexample, memory 4704 (e.g., database 4705) may store selectiveconditioning functions associated with particular objects.

In some embodiments, selective conditioning may include attenuation orsuppressing one or more audio signals not associated with soundemanating object 4710, such as sounds 4721 and 4722, which may emanatefrom other objects within the environment (e.g., sound emanating object4711), or may be background noise. Similar to amplification of sound4720, attenuation of sounds may occur through processing audio signals,or by varying one or more parameters associated with microphone 4702 todirect focus away from sounds not associated with sound emanating object4710.

Where more than one sound emanating object is detected, hearing aidsystem 4700 may selectively condition sounds associated with the soundemanating objects relative to each other. For example, the at least onesound emanating object may include a first sound emanating object (e.g.,sound emanating object 4710) and a second sound emanating object (e.g.,sound emanating object 4711). Selective conditioning may includeattenuating a first audio signal associated with the first soundemanating object; and amplifying a second audio signal associated withthe second sound emanating object. Similarly, selective conditioning mayinclude changing a tone of a first audio signal associated with thefirst sound emanating object; and avoiding from changing a tone of asecond audio signal associated with the second sound emanating object.Accordingly, the audio signal associated with the first sound emanatingobject may be more perceptible to user 100. Where the sound emanatingobjects are individuals, selective conditioning may include changing arate of speech associated with the first individual and avoid fromchanging a rate of speech associated with the second individual. Forexample, processor 4703 may add short pauses between words associatedwith the first individual in order to make the audio more intelligible.Various other forms of selective conditioning may also be performed toimprove the presentation of the audio signal to user 100.

Hearing aid system 4700 may perform other actions, such as presentingthe determined identity of sound emanating object 4710 to user 100. Theidentity may be presented in various ways. In some embodiments, hearingaid system 4700 may audibly present the identification of the object tothe user, for example, through hearing interface device 1710, computingdevice 120, or various other devices. Hearing aid system 4700 may readthe name of the detected object to the user. Accordingly, hearing aidsystem may access one or more speech-to-text algorithms or softwarecomponents for presenting a name of an object in database 4705. In otherembodiments, prerecorded names of the objects may be stored in memory4704. Where the sound emanating object is an individual, hearing aidsystem 4700 may present the name of the individual to the user and/orother information associated with the individual (e.g., a relationshipto the individual, an age of the individual, names of other individualsassociated with the individual, a title of the individual, etc.).

Hearing aid system 4700 may also present the determined identity ofsound emanating object 4710 to user 100 visually. FIG. 48 is anillustration showing an exemplary device displaying the name of a soundemanating object consistent with the present disclosure. As shown inFIG. 48, hearing aid system 4700 may display information about soundemanating object 4710 on a display of device 4801. In some embodiments,device 4801 may be a paired wearable device, such as a mobile phone,tablet, personal computer, smart watch, heads up display (HUD), or thelike. In embodiments where sound emanating device 4710 is an individual,the at least one action performed by hearing aid system 4700 may includecausing a name 4810 of the individual to be shown on the display.Various other information may also be presented on the display. Forexample, device 4801 may display an image 4811 of the object orindividual, as shown in FIG. 48. Where sound emanating object is anindividual, hearing aid system 4700 may display various otheridentification information associated with the individual (e.g., a phonenumber, address, title, company, relationship, age, etc.). The displaymay also include other functionality associated with the individual,such as contacting the individual (e.g., by phone, email, SMS, etc.),access an account associated with the individual (e.g., a social mediapage, file sharing account or location, etc.), or the like. In someinstances, the display may also include functionality for confirming orediting the identification of sound emanating object 4710, for example,to improve a trained neural network or other machine learning system, asdescribed above.

FIG. 49 is a flowchart showing an exemplary process 4900 for using voiceand visual signatures to identify objects consistent with disclosedembodiments. Process 4900 may be performed by hearing aid system 4700,for example by processor 4703. As described above, processor 4703 maycorrespond to one or more other processors described in detail above,including processors 210, 210 a and/or 210 b. Accordingly, process 4900may be performed by a processor associated with a wearable cameradevice, such as apparatus 110. Some or all of process 4900 may beperformed by processors associated with other components, such ascomputer device 120, server 250, or other devices. As described above,hearing aid system 4900 may include memory 4704 configured to store adatabase (e.g., database 4705) of reference visual characteristics andreference voiceprints corresponding to a plurality of objects. Processor4703, or the processor performing various steps of process 4900 mayaccess memory 4704.

In step 4910, process 4900 may include receiving image data and audiosignals associated with at least one sound emanating object. Forexample, step 4910 may include receiving a plurality of images capturedby a wearable camera, wherein at least one of the plurality of imagesdepicts at least one sound emanating object in an environment of a user.The images may be captured, for example, by wearable camera 4701 and mayinclude a representation of sound emanating object 4710. The images maybe received by processor 4703. Step 4910 may further include receivingaudio signals acquired by a wearable microphone, wherein the audiosignals are representative of one or more sounds emanating from the atleast one sound emanating object. For example, processor 4703 mayreceive audio signals from microphone 4702 which may represent sound4720 emanating from sound emanating object 4710. The audio signals maybe received concurrently with the captured images, or may be receivedlater during process 4900, for example, after an identification of soundemanating object 4710 has been made based on the captured images.

In step 4920, process 4900 may include analyzing at least one of thereceived plurality of images to determine one or more visualcharacteristics associated with the at least one sound emanating object.For example, processor 4703 may use one or more image recognitiontechniques to extract features from the image that are associated withsound emanating object 4710. The extracted features may be analyzed todetermine the visual characteristics, which may include a color, shape,arrangement, size, or other characteristic of the object. The visualcharacteristics may be indicative of the type of an object, such aswhether the object is an individual or an inanimate object, aclassification of the object, etc. In some instances, sound emanatingobject 4710 may be an individual. Accordingly, step 4920 may includedetermining the visual characteristics based on a facial analysis of animage of the individual. Accordingly, processor 4703 may identify facialfeatures on the face of the individual, such as the eyes, nose,cheekbones, jaw, or other features. Processor 4703 may use one or morealgorithms for analyzing the detected features, such as principalcomponent analysis (e.g., using eigenfaces), linear discriminantanalysis, elastic bunch graph matching (e.g., using Fisherface), LocalBinary Patterns Histograms (LBPH), Scale Invariant Feature Transform(SIFT), Speed Up Robust Features (SURF), or the like.

In step 4930, process 4900 may include identifying (or attempting toidentify) within the database in view of the one or more visualcharacteristics, the at least one sound emanating object and determine adegree of certainty of identification. Accordingly, process 4900 mayfurther include accessing a database of reference visual signatures andreference voice signatures corresponding to a plurality of objects. Asdescribed above, processor 4703 may access database 4705, which maystore a plurality of visual characteristics, a plurality of objects, andassociations between the visual characteristics and the objects.Processor 4703 may attempt to match the visual characteristicsdetermined in step 4920 to visual characteristics within database 4705.In some embodiments, as described above, the at least one soundemanating object may include a first sound emanating object and a secondsound emanating object, and step 4930 may further comprise usingdetermined visual characteristics of the first sound emanating object toidentify the second sound emanating object. Processor 4703 may determinea confidence score corresponding to a degree of certainty that the soundemanating object represented in the captured images corresponds to oneor more objects in database 4705. In some embodiments, step 4730 mayinclude generating a confidence score for more than one object indatabase 4705 and identifying sound emanating object 4710 as the objectin database 4710 corresponding to the highest confidence score.

In some instances, the at least one sound emanating object may beidentified based on the visual characteristics alone. In some instances,however, process 4900 may further include identifying the at least onesound emanating object based on audio signals associated with the soundemanating object(s). Accordingly, process 4900 may include a step 4935of determining whether identification based on the visualcharacteristics is sufficient. For example, step 4935 may comprisecomparing the confidence score determine in step 4930 with a certainthreshold. Where the confidence scores are represented as a percentage(with 100% representing a maximum confidence), for example, thethreshold may be an intermediate value (e.g. 40%, 50%, 60%, 70%, etc.).The threshold may be higher or lower depending on the use of the system.In some embodiments the threshold may vary based on various otherfactors or settings, for example, based on the type of objectidentified, an image quality, an importance value associated withcorrectly identifying the object, a time of day, a threshold set by auser, a threshold set by an administrator, etc.). If the confidencescore exceeds the threshold, process 4900 may proceed to step 4960, asindicated in FIG. 49. If the confidence score is below the threshold,however, process 4900 may proceed to step 4940. The outcome of step 4935may be determined by other factors besides the confidence score. Forexample, a user or administrator may change a setting to always proceedto step 4960 or 4940. In other embodiments the determination may bebased on other factors, such as a type of the sound emanating object(e.g., whether the object is an individual, etc.) or an importance value(e.g., if hearing aid system is identifying an oncoming vehicle, etc.).

In step 4940, process 4900 may include analyzing received audio signalsto determine a voiceprint of the at least one sound emanating object. Asdiscussed above, with respect to step 4910, step 4940 may include a stepof receiving audio signals acquired by a wearable microphone if theaudio signals have not yet been received. The audio signals may berepresentative of one or more sounds emanating from the at least onesound emanating object. Processor 4703 may analyze the received audiosignals to identify a voiceprint of the sound emanating object. Ininstances where the at least one sound emanating object is anindividual, step 4940 may include determining the voiceprint based onaudio analysis of a recording of the individual. For example, processor4703 may use one or more voice recognition algorithms, such as HiddenMarkov Models, Dynamic Time Warping, neural networks, or othertechniques, to recognize the voice of the individual. The determinedvoiceprint may include characteristics of the individual, such as anaccent, age, gender, vocabulary, or the like.

In step 4950, process 4900 may include identifying the at least onesound emanating object based on the visual characteristics and thedetermined voiceprint. For example, processor 4703 may access database4705, which may store voiceprint data associated with a plurality ofobjects. Processor 4703 may be configured to determine a match betweenthe voiceprint determined in step 4940 and the voiceprint data stored indatabase 4705. In some embodiments the identification of the at leastone sound emanating object using the determined visual characteristics(e.g., in step 4930) results in a group of candidate objects, and theidentification of the at least one sound emanating object includesselecting one of the group of candidate objects based on the voiceprint.In other embodiments, the voiceprint data may be used to identifycandidate objects independently and compare the candidate objects tothose identified in step 4930. In some embodiments, as described above,the at least one sound emanating object may include a first soundemanating object and a second sound emanating object, and step 4930 mayfurther comprise using determined visual characteristics of the firstsound emanating object to identify the second sound emanating object.Step 4950 may further include determining a confidence score associatedwith the identification based on the voiceprint. In some embodiments,the confidence score may be cumulative, representing a confidence basedon both the visual characteristic identification in step 4930 and thevoiceprint identification in step 4950. In other embodiments, avoiceprint confidence score may be determined separately.

In step 4955, process 4950 may include reassessing the identification ofthe at least one sound emitting object. Similar to step 4935, step 4955may comprise comparing the confidence score from step 4950 with apredetermined threshold. Threshold may be the same threshold describedabove with reference to step 4935 or may be a different threshold. Forexample, a confidence score based on a combined analysis under steps4930 and 4950 may be subject to a higher confidence score threshold thanbased on step 4930 alone. The threshold value and the determinationunder step 4955 generally may be based on other factors as describedabove with respect to step 4935. If the confidence score exceeds thethreshold, process 4900 may proceed to step 4960. If the confidencescore does not meet the threshold value, however, process 4900 mayreturn to step 4910. For example, hearing aid system 4700 may determinethat the object cannot be identified based on the received images andaudio signals and may obtain additional images and/or audio signals tocomplete the identification. Process 4900 may include other steps, suchas sending a notification to a user indicating the identificationfailed, or the like.

In step 4960, process 4900 may comprise initiating at least one actionbased on an identity of the at least one sound emanating object. Asdescribed above, the at least one action may include causing selectiveconditioning of at least one audio signal associated with the at leastone sound emanating object. The at least one action may further includecausing transmission of the at least one conditioned audio signal to ahearing interface device configured to provide sounds to an ear of theuser, such as hearing interface device 1710. For example, the selectiveconditioning may include varying a tone, volume, or rate of speech ofthe audio signal, as discussed in greater detail above. In someinstances, the at least one sound emanating object includes a firstsound emanating object and a second sound emanating object, and causingselective conditioning of the at least one audio signal may includeattenuating a first audio signal associated with the first soundemanating object and amplifying a second audio signal associated withthe second sound emanating object. The selective conditioning mayfurther include changing a tone of a first audio signal associated withthe first sound emanating object and avoiding hanging a tone of a secondaudio signal associated with the second sound emanating object. In someinstances, the at least one sound emanating object may include a firstindividual and a second individual, and causing selective conditioningof the at least one audio signal may include changing a rate of speechassociated with the first individual and avoiding changing a rate ofspeech associated with the second individual.

In some embodiments, the at least one action may include storing in theat least one memory device information relating to an encounter with theat least one sound emanating object, as described in greater detailabove. The stored information may be used for updating the visualcharacteristics and/or voiceprint of the at least one sound emanatingobject in database 4705. For example, the stored information may be usedto ensure that database 4705 is accurate and/or up to date, as discussedin greater detail above.

In some embodiments, where the at least one sound emanating object(e.g., sound emanating object 4710) is an individual, the at least oneaction may include causing a name of the individual to be shown on adisplay, as discussed above in reference to FIG. 48. The display may beassociated with a paired wearable device (e.g., device 4801), such as amobile phone, smartwatch, or other mobile device. Other information orfunctionality may also be displayed for user 100, as discussed in detailabove.

Selective Input for a Hearing Aid Based on Image Data

Consistent with the disclosed embodiments, a hearing aid system mayselectively condition audio signals from sound emanating objects withinthe environment of a user. The hearing aid system may access a databasestoring information about various sound emanating objects and mayselectively condition audio from the sound emanating objects based onthe information stored in the database. As one example, the hearing aidsystem may determine a relative rank or importance of the various soundemanating objects and selectively condition audio signals associatedwith the sound emanating objects based on the relative rank orimportance. The hearing aid system may also selectively condition audiosignals from the sound emanating objects based on the context, forexample based on the location of the user.

The hearing aid system of the present disclosure may correspond tohearing aid system 4700, described above with respect to FIG. 47A. Forexample, the hearing aid system may include at least one wearable camera4701, at least one microphone 4702, at least one processor 4703, and atleast one memory 4704. While the hearing aid system for selectivelycondition audio signals from sound emanating objects is described inreference to hearing aid system 4700 throughout the present disclosure,it is understood that the hearing aid system may be separate and/ordifferent from hearing aid system 4700. For example, the hearing aidsystem may include additional or fewer components than those shown inFIG. 47A. Further, as discussed above, the components shown in FIG. 47Amay be housed in a single device or may be contained in one or moredifferent devices.

As discussed above, wearable camera 4701 may be configured to captureone or more images from the environment of user 100. In someembodiments, wearable camera 4701 may be included in a wearable cameradevice, such as apparatus 110. For example, wearable camera 4701 may becamera 1730, as described above, which may also correspond to imagesensor 220. Microphone 4702 may be configured to capture sounds from theenvironment of user 100. In some embodiments, camera 4701 and microphone4702 may be included in the same device. Microphone 4702 may be includedin a wearable camera device, such as apparatus 110. For example,apparatus 110 may comprise microphone 1720, as described with respect toFIG. 17B, which may be configured to determine a directionality ofsounds in the environment of user 100. Apparatus 110 may be worn by user100 in various configurations, including being physically connected to ashirt, necklace, a belt, glasses, a wrist strap, a button, or otherarticles associated with user 100. In some embodiments, one or moreadditional devices may also be included, such as computing device 120.Accordingly, one or more of the processes or functions described hereinwith respect to hearing aid system 4700 or processor 4703 may beperformed by computing device 120 and/or processor 540.

Processor 4703 may be configured to receive and process images and audiosignals captured by wearable camera 4701 and microphone 4702. Asdiscussed above, processor 4703 may be associated with apparatus 110,and thus may be included in the same housing as wearable camera 4701 andmicrophone 4702. For example, processor 4703 may correspond toprocessors 210, 210 a or 210 b, as described above with respect to FIGS.5A and 5B. In other embodiments, processor 4703 may be included in oneor more other devices, such as computing device 120, server 250 (FIG. 2)or various other devices. In such embodiments, processor 4703 may beconfigured to receive data remotely, such as images captured by wearablecamera 4701 and audio signals captured by microphone 4702.

Memory 4704 may be configured to store information associated with soundemanating objects in the environment of user 100. Memory 4704 may be anydevice capable of storing information about one or more objects, and mayinclude a hard drive, a solid state drive, a web storage platform, aremote server, or the like. Memory 4704 may be located within apparatus110 (e.g., within memory 550) or external to apparatus 110. In someembodiments, memory 4704 may further include a database, such asdatabase 5020, which is described in detail below.

Apparatus 110 may also communicate with a hearing interface device wornby user 100. For example, the hearing aid device may be hearinginterface device 1710, as shown in FIG. 17A. As described above, hearinginterface device 1710 may be any device configured to provide audiblefeedback to user 100. Hearing interface device 1710 may be placed in oneor both ears of user 100, similar to traditional hearing interfacedevices. Hearing interface device 1710 may be of various styles,including in-the-canal, completely-in-canal, in-the-ear, behind-the-ear,on-the-ear, receiver-in-canal, open fit, or various other styles.Hearing interface device 1710 may include one or more speakers forproviding audible feedback to user 100, a communication unit forreceiving signals from another system, such as apparatus 110,microphones for detecting sounds in the environment of user 100,internal electronics, processors, memories, etc. Hearing interfacedevice 1710 may correspond to feedback outputting unit 230 or may beseparate from feedback outputting unit 230 and may be configured toreceive signals from feedback outputting unit 230.

In some embodiments, hearing interface device 1710 may comprise a boneconduction headphone 1711, as shown in FIG. 17A. Bone conductionheadphone 1711 may be surgically implanted and may provide audiblefeedback to user 100 through bone conduction of sound vibrations to theinner ear. Hearing interface device 1710 may also comprise one or moreheadphones (e.g., wireless headphones, over-ear headphones, etc.) or aportable speaker carried or worn by user 100. In some embodiments,hearing interface device 1710 may be integrated into other devices, suchas a Bluetooth™ headset of the user, glasses, a helmet (e.g., motorcyclehelmets, bicycle helmets, etc.), a hat, etc. Hearing interface device1710 may be configured to communicate with a camera device, such asapparatus 110. Such communication may be through a wired connection, ormay be made wirelessly (e.g., using a Bluetooth™, NFC, or forms ofwireless communication). Accordingly, hearing interface device 1710 mayinclude a receiver configured to receive at least one audio signal andan electroacoustic transducer configured to provide sounds from the atleast one audio signal to an ear of the user.

FIG. 50A is a schematic illustration showing examples of sound emittingobjects that may be identified in an environment 5000 of a userconsistent with the present disclosure. As discussed above, the soundemanating objects may include any objects capable of emitting soundsthat are perceptible to user 100 or apparatus 110. In some instances,sound emanating objects may be a person, such as individuals 5010 and5011 shown in FIG. 50A. In other embodiments the sound emanating objectsmay be a device, such as television 5012, shown in FIG. 50A. Soundemanating objects may include other devices, such as a radio, a speaker,a television, a mobile device (e.g., a mobile phone, tablet, etc.), acomputing device (e.g., personal computer, desktop computer, laptop,gaming console, etc.), vehicles, alarms, or any other device capable ofemitting sounds. Sound emanating objects may also include other objects,such as pets, animals, insects, natural features (e.g., streams, trees,etc.), inanimate objects, weather-related objects, or any other objectsor portions of an object that may emanate sounds.

FIG. 50B is an illustration of an example database 5020 storinginformation associated with sound emanating objects consistent with thepresent disclosure. Database 5020 may be maintained on any memoryassociated with hearing aid system 4700, such as memory 4704. Database4705 may correspond to database 4705, described above, or may be aseparate database. In some embodiments, database 5020 may be locatedseparately from hearing aid system 4700, for example on a remote deviceor server, and may be accessible by hearing aid system 4700. As shown inFIG. 50B, database 5020 may store visual characteristics of one or moresound emanating objects. The visual characteristics may include featuresor attributes of the sound emanating objects that may be detected byhearing aid system 4700. For example, the visual characteristics mayinclude a size, color, shape, pattern, or other visual features of theassociated sound emanating object. Visual characteristics may includefacial features used to identify a particular individual.

Database 5020 may include other information about the sound emanatingobject, such as a name, type, relationship, level of importance,voiceprint data, and/or rules for audio conditioning. Where the soundemanating object is an individual, the name of the sound emanatingobject may be associated with the individual's name. The relationshipstored in database 5020 may define a relationship between the individualand user 100, such as whether the individual is a friend, colleague,family relative, acquaintance, or any other forms of relationships thatmay be defined. For example, as shown in FIG. 50B, individual CindyMoore may be a colleague of user 100, where individual Raj Polar may bea friend of user 100. In some embodiments, more specific relationshipsmay be defined, such as identifying a co-worker as a manager of theuser, identifying a family member as the user's father, identifying afriend as close friend, etc. In some embodiments, database 5020 may beassociated with a list of contacts of user 100, a social networkplatform (e.g., Facebook™, LinkedIn™, Instagram™, etc.), or variousother associated lists or databases, and may be configured to determinea relationship based on data received from the lists or databases.

The sound emanating object may also be a device or other object, asdescribed above. In some instances, the name of the sound emanatingobject may be a generic name of the device (e.g., laptop, television,phone, etc.) In some embodiments, hearing aid system 4700 may recognizea particular device, rather than just a general device type.Accordingly, the name of the sound emanating object stored in database5020 may be specific to the detected device. For example, the name mayidentify the owner of the device, (e.g., “my phone,” “Terri's laptop,”etc.). In some embodiments, the name may also include a serial number orother unique identifier of the device. Similarly, the relationship ofthe sound emanating object may indicate whether the sound emanatingobject is associated with user 100 in some way.

Database 5020 may further store information pertaining to selectiveaudio conditioning of the sound emanating object. For example, the levelof importance may rank the sound emanating objects in database 5020relative to each other. In some embodiments, each device may be uniquelyranked relative to each of the other sound emanating objects in thedatabase. In other embodiments, the sound emanating objects may beranked on a scale (e.g., 1-5, 1-10, 1-100, etc.), as a percentage, basedon predefined ranking levels (e.g., “high importance,” “low importance,”etc.) or any other suitable ranking method. In some embodiments, theranking may be based on the relationship to the user. For example,family members of user 100 may be given a higher importance ranking thanacquaintances of user 100. Similarly, a manager or boss of user 100 maybe given a higher importance ranking than a peer of user 100. Database5020 may also store specific audio conditioning rules associated withthe sound emanating object. For example, as shown in FIG. 50B, the rulesmay include a predefined conditioning parameter to be applied to anaudio signal associated with the sound emanating object, such aschanging a pitch or volume of the audio signal. The conditioningparameter may be absolute (e.g., a set volume level, +10% volume, etc.)or may be defined relative to other sounds in the environment (e.g.,increase volume relative to other sounds). In some embodiments, the rulemay be associated with one or more other parameters, such as therelationship to the user. For example, the rule may apply to all familymembers of user 100 or may apply to individuals of a certain level ofimportance. In some embodiments, the rules may further includecontext-based conditions, for example, based on current or previousactions of user 100, the environment of user 100 or any othercontext-based rules. Referring to the examples shown in FIG. 50B,hearing aid system 4700 may be configured to mute a television when user100 is not looking at it or increase the volume of an individual whenthey are meeting outside. Accordingly, hearing aid system 4700 may beconfigured to determine an environment of the user, for example, basedon analyzing other objects in captured images, analyzing captured audio,using global positioning system (GPS) data, or the like. Other audioconditioning methods are described in greater detail below.

In some embodiments, database 5020 may also store voiceprint dataassociated with sound emanating objects. The voiceprint data may beunique to the particular sound emanating object that it is associatedwith (similar to the voiceprint data in database 4705, described above).Accordingly, the voiceprint may be suitable for identifying the soundemanating object. For example, processor 4703 may identify a soundemanating object, such as an individual, through the visualcharacteristics described above and may retrieve information associatedwith the sound emanating object from database 5020. In some embodiments,the information may include the voiceprint, which may be used forfurther identifying the sound emanating object, or the like. In someinstances, the voiceprint information for a particular sound emanatingobject may include a set of reference voiceprints. For example, a firstvoiceprint of a specific individual may be associated with a scenariowhere the specific individual is standing next to the user, and a secondvoiceprint of the specific individual may be associated with a scenariowhere the specific individual is talking through a communication device.

In some embodiments, the information stored in database 5020 may bedesignated and/or modified by user 100 or another individual (e.g., acaretaker, administrator, etc.). For example, user 100 may manually addto or edit the sound emanating objects in database 5020, for examplethrough a user interface, such as computing device 120. User 100 maydefine the name of the sound emanating object, the classification type,the relationships, the level of importance, and/or the rules for audioconditioning. In some embodiments, database 5020 may be built and/ormodified through an automated process. For example, hearing aid system4700 may be configured to learn one or more properties or valuesassociated with a sound emanating object based on the interaction ofuser 100 with the sound emanating object. If user 100 continuallyincreases the volume of hearing interface device 1710 when interactingwith a particular sound emanating object, hearing aid system 4700 mayautomatically include a rule to increase the volume of audio signalsassociated with that sound emanating object. As another example, user100 may more frequently look at a particular sound emanating objectrelative to other sound emanating objects and hearing aid system 4700may assign a level of importance, relationship, rule for audioconditioning, or another property based on the behavior of user 100.

FIG. 51A is a schematic illustration showing an example environment 5100for selectively conditioning audio signals consistent with the presentdisclosure. Environment 5100 of user 100 may include one or more soundemanating objects, as discussed above. For example, environment 5100 mayinclude sound emanating objects 5110 and 5111, which may be individuals,and sound emanating object 5512, which may be a device.

Hearing aid system 4700 may be configured to receive images and/or audiosignals associated with sound emanating objects 5110, 5111, and 5112.For example, wearable camera 4701 may be included in apparatus 110, wornby user 100. Wearable camera 4701 may capture an image including arepresentation of sound emanating object 5110 within the environment ofuser 100. Processor 4703 may receive a plurality of images captured bywearable camera 4701 and analyze the images to determine visualcharacteristics of sound emanating object 5110. Such visualcharacteristics may include any features of the object represented inthe image. For example, the visual characteristics may include a color,shape, size, type, or the like, which may correspond to the visualcharacteristic types stored in database 5020. Accordingly, processor4703 may use one or more image recognition techniques or algorithms todetect features of sound emanating object 5110. For example, processor4703 may identify one or more points, edges, vertices or other featuresof the object. Where sound emanating object 5110 is an individual,processor 4703 may further determine the visual characteristics based ona facial analysis or face recognition of an image of the individual.Accordingly, processor 4703 may use one or more algorithms for analyzingthe detected features, such as principal component analysis (e.g., usingeigenfaces), linear discriminant analysis, elastic bunch graph matching(e.g., using Fisherface), Local Binary Patterns Histograms (LBPH), ScaleInvariant Feature Transform (SIFT), Speed Up Robust Features (SURF), orthe like. Similar feature recognition techniques may be used fordetecting features of inanimate objects as well, such as sound emanatingobject 5112.

In addition to identifying sound emanating objects, hearing aid system4700 may also determine a context of the sound emanating objects.Accordingly, processor 4703 may be configured to analyze other featuresor objects within the captured images. For example, objects such astrees, flowers, grass, buildings, etc. may indicate that user 100 isoutside. Other objects, such as chairs, desks, computer screens,printers, etc. may indicate that user 100 is in an office environment.In some embodiments, processor 4703 may associate particular objects orgroups of objects with a particular environment of user 100. Forexample, processor 4703 may recognize one or more objects to determinethat user 100 is in a particular room, such as a living room of user100, a particular office or conference room, etc. This contextualinformation may be used for selectively conditioning audio signalsassociated with a sound emanating object, as described in further detailbelow.

Processor 4703 may further be configured to receive audio signalsassociated with sound emanating objects in the environment of user 100.The audio signals may be representative of one or more sounds emanatingfrom the sound emanating object. For example, sound emanating objects5110, 5111, and 5112 may emanate sounds 5120, 5121, and 5122,respectively, as shown in FIG. 51A. Hearing aid system 4700 may beconfigured to capture sounds 5120, 5121, and 5122 (e.g., throughmicrophone 4702) and convert them to an audio signal to be processed byprocessor 4703. In instances where the sound emanating object is anindividual, such as sound emanating object 5110, sound 5120 may be avoice of the individual. Where the sound emanating object is a device orother object, such as sound emanating object 5112, sound 5122 may be anoutput of the device, such as sound from a television, mobile phone, orother device, a sound produced by a vehicle, etc.

In some embodiments, processor 4703 may be configured to determine avoiceprint of the sound emanating object. The voiceprint may bedetermined according to any of the methods discussed above with respectto FIG. 47B. For example, processor 4703 may use one or more voicerecognition algorithms (e.g., Hidden Markov Models, Dynamic TimeWarping, neural networks, or other techniques) to recognize the voice ofthe individual. The determined voiceprint may include variouscharacteristics associated with the individual, such as an accent of theindividual, the age of the individual, a gender of the individual, orthe like. While the voiceprint may represent a voice pattern of anindividual, the term voiceprint should be interpreted broadly to includeany sound pattern or feature that may be used to identify a soundemanating object.

Hearing aid system 4700 may be configured to selectively condition thesound received from one or more sound emanating objects. In someembodiments, conditioning may include changing a tone of one or moreaudio signals corresponding to sound 5120 to make the sound moreperceptible to user 100. User 100 may have lesser sensitivity to tonesin a certain range and conditioning of the audio signals may adjust thepitch of sound 5120. For example, user 100 may experience hearing lossin frequencies above 10 kHz and processor 4703 may remap higherfrequencies (e.g., at 15 kHz) to 10 kHz. In some embodiments, processor4703 may be configured to receive information about the user's hearingcapabilities and cause the conditioning of at least one audio signal isbased on the user's hearing capabilities.

In some embodiments processor 4703 may be configured to change a rate ofspeech associated with one or more audio signals. Processor 4703 may beconfigured to vary the rate of speech of sound emanating object 5110 tomake the detected speech more perceptible to user 100. Selectiveconditioning may also include adding one or more spaces or pauses withinthe audio signal. For example, the sound emanating object may include anindividual saying a sentence and causing the conditioning of at leastone audio signal includes adding at least one space between words in thesentence to make the sentence more intelligible. Accordingly, ratherthan hearing the spoken sentence at 1× speed, user 100 may hear thesentence at an increased speed (e.g., 1.1×, 1.5×, 2.0×, 2.5×, etc.) andthe space between each word may be increased accordingly. Similarly, thespacing between sentences may be increased, giving user 100 more time tointerpret or digest each sentence.

In some embodiments, hearing aid system 4700 may selectively conditionthe audio signals based on information about the identified soundemanating object retrieved from database 5020. For example, processor4703 may receive an audio signal associated with sound 5120 from soundemanating object 5110. Based on the identification of sound emanatingobject 5110, processor 4703 may retrieve information about the soundemanating object from database 5020. For example, identifying the soundemanating object may include determining a type of the sound emanatingobject, and processor 4700 may further be programmed to cause selectiveconditioning of the audio signal based on the determined type of the atleast one sound emanating object. In another embodiment, the retrievedinformation may be associated with a pre-existing relationship of user100 with the sound emanating object, and the at least one processor maybe further programmed to cause the selective conditioning of the atleast one audio signal based on the pre-existing relationship. In someembodiments, the selective conditioning may also be performed based on acontextual situation associated with user 100. The contextual situationmay be determined by analysis of one or more images captured from acamera device, such as wearable camera 4701. The conditioning of a soundemanating object determined through database 5020 may be different basedon the context. As an illustrative example, if the sound emanatingobject is a crying baby, the selective conditioning may includeamplifying the volume of the audio signal associated with the baby ifuser 100 is at home. Conversely, if hearing aid system 4700 determinesuser 100 is on an airplane, the selective conditioning may includemuting the audio signal associated with the crying baby.

Where more than one sound emanating object is detected, hearing aidsystem 4700 may selectively condition sounds associated with the soundemanating objects relative to each other. In the example scenario shownin FIG. 51A, sound emanating objects 5110 and 5111 may comprise twoindividuals. Processor 4703 may be programmed to cause a first selectiveconditioning of audio signals associated with a first individual (e.g.,sound emanating object 5110) based on retrieved information associatedwith the first individual, and cause a second selective conditioningdifferent from the first selective conditioning of audio signalsassociated with second individual (e.g., sound emanating object 5111)based on retrieved information associated with the second individual.For example, the first individual may be difficult to understand andprocessor 4703 may increase the volume or vary the pitch of the audiosignal associated with the first individual. Processor 4703 maydetermine that the audio signal associated with the second individual isof lesser importance (e.g., based on a relationship, importance level,etc.) and may decrease the volume associated with the second individual.As another example, processor 4703 may analyze a plurality of images toidentify an individual (e.g., sound emanating object 5110) that isspeaking and a sound emanating object that generates background noises(e.g., sound emanating object 5112). Processor 4703 may be configured toseparate sounds generated by the individual from the background noises.Accordingly, causing selective conditioning of audio signals may includeattenuating audio signals associated with the sound emanating objectrelative to the audio signals associated with the individual. Forexample, if the sound emanating object is a television, such as soundemanating object 5012, selective conditioning of audio signals mayinclude reducing the volume of the television or muting it completely.

FIG. 51B is a schematic illustration showing another example environment5101 for selectively conditioning audio signals consistent with thepresent disclosure. In this scenario user 100 may be wearing apparatus110 and may be in the presence of sound emanating objects 5110 and 5111,which may be individuals, as described above. Environment 5101 mayinclude a third sound emanating object 5113, which may also be anindividual. Processor 4703 may be configured to selectively conditionaudio signals associated with sound emanating objects 5110, 5111, and5113 based on the interactions between sound emanating objects 5110,5111, and 5113 and/or user 100. In the scenario shown in FIG. 51B,processor 4703 may identify a first individual (e.g., sound emanatingobject 5110) talking to user 100 and a second individual (e.g., soundemanating object 5111) talking to a third individual (e.g., soundemanating object 5113). Accordingly, processor 4703 may amplify audiosignals associated with the first individual and attenuate audio signalsassociated with the second individual. In another scenario, processor4703 may identify a group of individuals listening to a specificindividual and may be programmed to amplify audio signals from thespecific individual.

As discussed above, database 5020 may include one or more voiceprintsassociated with a particular sound emanating object. Processor 4703 mayinclude instructions to receive a reference voiceprint associated with asound emanating object that has been identified based on the visualcharacteristics. Accordingly, processor 4703 may be configured to usethe plurality of images and the reference voiceprint to identify the atleast one sound emanating object and to cause the conditioning of atleast one audio signal based on predefined settings associated with anidentity of the at least one sound emanating object. The predefinedsettings may correspond to information stored in database 5020,including rules for selectively conditioning audio, a level ofimportance, a relationship with user 100, or various other parametersthat may or may not be shown in FIG. 50B. For example, based on thevoiceprint data, processor 4703 may determine that certain sounds (e.g.,a siren, a baby crying, etc.) should be heard but may reduce the volumeof background noises (e.g., an air conditioning unit, traffic, noiseoffice mates, etc.). In some embodiments, processor 4703 may further usethe voiceprints to separate audio signals associated with various soundemanating objects. For example, each sound emanating object isassociated with a unique voiceprint and processor 4703 may usevoiceprints of the sound emanating objects to separate sounds generatedby a first sound emanating object and sounds generated by a second soundemanating object. Causing the conditioning of at least one audio signalmay include attenuating audio signals associated with the second soundemanating object relative to the audio signals associated with the firstsound emanating object.

FIG. 52 is a flowchart showing an exemplary process 5200 for modifyingsounds emanating from objects in an environment of a user consistentwith the disclosed embodiments. Process 5200 may be performed by ahearing aid system (e.g., hearing aid system 4700), which may include atleast one processor (e.g., processor 4703) programmed to perform thesteps described below. Processor 4703 may correspond to one or moreother processors described in detail above, including processors 210,210 a and/or 210 b. Accordingly, process 5200 may be performed by aprocessor associated with a wearable camera device, such as apparatus110. Some or all of process 5200 may be performed by processorsassociated with other components, such as computing device 120, server250, and/or other devices. As described above, hearing aid system 4700may access a database (e.g., database 5020), which may containinformation for selectively conditioning audio for one or more soundemanating objects. The database may be internal to the hearing aidsystem (e.g., stored within memory 4704) or may be external (e.g.,accessed via a network connection, a short-range wireless connection,etc.). The hearing aid system may further comprise at least one wearablecamera (e.g., wearable camera 4701) and at least one wearable microphone(e.g., microphone 4702). In some embodiments, the wearable camera, thewearable microphone, and the at least one processor may be included in acommon housing (e.g., in apparatus 110). In other embodiments, thewearable camera, the wearable microphone, and the at least one processormay be distributed among multiple housings. For example, the wearablecamera and the wearable microphone are included in a first housing andthe at least one processor is included in a second housing separate fromthe first housing.

In step 5210, process 5200 may include receiving a plurality of imagescaptured by the wearable camera. For example, step 5210 may includereceiving a plurality of images captured from the environment of theuser (e.g., user 100) by the wearable camera. Accordingly, the pluralityof images may depict objects in an environment of a user. The pluralityof images may include a representation of a sound emanating object, suchas sound emanating object 5110. In step 5220, process 5200 may includereceiving audio signals acquired by the wearable microphone. The audiosignals may be representative of sounds emanating from the objectsdepicted in the plurality of images received in step 5210. For example,processor 4703 may receive audio signals from microphone 4702 which mayrepresent sound 5120 emanating from sound emanating object 5110.

In step 5230, process 5200 may include analyzing the plurality of imagesto identify at least one sound emanating object in the environment ofthe user. For example, processor 4703 may use one or more imagerecognition techniques to extract features from the image that areassociated with sound emanating object 5110. In some instances, the atleast one sound emanating object may include an individual and,accordingly, step 5230 may include performing a facial analysis or facerecognition of an image of the individual. In some embodiments,identifying the at least one sound emanating object may includedetermining a type of the at least one sound emanating object. Forexample, processor 4703 may determine whether sound emanating object5510 is a mechanical machine or device, a speaker, an individual, ananimal, an inanimate object, a weather-related object, or the like.

In step 5240, process 5200 may include retrieving, from a database,information about the at least one sound emanating object. For example,processor 4703 may access database 5020 storing information about one ormore sound emanating objects. The stored information may refer to aclass of sound emanating objects (e.g., televisions), or may refer to aspecific sound emanating object (e.g., a specific person, the user'sphone, etc.). As described above in reference to FIG. 50B, database 5020may store information including visual characteristics of the object, aname of the object, a type of object, a relationship of the object tothe user, a level of importance of the object, voiceprint dataassociated with the object, a rule of audio conditioning for the object,or other information.

In step 5250, process 5200 may include causing, based on the retrievedinformation, selective conditioning of at least one audio signalreceived by the wearable microphone from a region associated with the atleast one sound emanating object. The region may be determined using thevarious methods described above (e.g., as shown in FIG. 20A). Forexample, the region may be determined based on a determined direction ofthe sound emanating object based on analysis of one or more of theplurality of images or audio signals. The range may be associated withan angular width about the direction of the sound emanating object(e.g., 10 degrees, 20 degrees, 45 degrees, etc.).

Various forms of conditioning may be performed on the audio signal, asdiscussed above. In some embodiments, conditioning may include changingthe tone or playback speed of an audio signal. For example, conditioningmay include changing a rate of speech associated with the audio signal.As discussed above, the at least one sound emanating object may includean individual saying a sentence and causing the conditioning of at leastone audio signal may include adding at least one space between words inthe sentence to make the sentence more intelligible. In someembodiments, the conditioning may include amplification of the audiosignal relative to other audio signals received from outside of theregion associated with the recognized individual. Amplification may beperformed by various means, such as operation of a directionalmicrophone configured to focus on audio sounds emanating from theregion, varying one or more parameters associated with the wearablemicrophone to cause the microphone to focus on audio sounds emanatingfrom the region, modifying one or more properties of the audio signal,or the like. The amplification may include attenuating or suppressingone or more audio signals received by the microphone from directionsoutside the region. As discussed above, selective conditioning maydepend on the preferences or hearing capabilities of the user. Forexample, the retrieved information (e.g., information received in step5240) may include information indicative of the user's hearingcapabilities and causing the conditioning of at least one audio signalmay be based on the user's hearing capabilities.

In some embodiments, identifying the at least one sound emanating object(e.g., in step 5230) may include determining a type of the at least onesound emanating object, and the at least one processor may be furtherprogrammed to cause the selective conditioning of the at least one audiosignal based on the determined type of the at least one sound emanatingobject. For example, the voice of an individual may be amplified whereasthe sound from a television may be reduced or muted. In otherembodiments, the retrieved information may be associated with apre-existing relationship of the user with the at least one soundemanating object, and the at least one processor may further beprogrammed to cause the selective conditioning of the at least one audiosignal based on the pre-existing relationship. For example, processor4703 may recognize that a sound emanating object is a phone belonging touser 100 and may amplify audio signals associated with the phonebelonging to user 100, but may not amplify (or may mute or attenuate)audio signals associated with other phones. Where the at least one soundemanating object includes a plurality of objects, processor 4703 mayapply a hierarchy of amplification for audio signals associated with theobjects. In such embodiments, the hierarchy of amplification may bebased on the pre-existing relationships.

Consistent with the present disclosure, processor 4703 may selectivelycondition audio associated with one sound emanating object relative toother sound emanating objects. For example, the at least one soundemanating object may include a plurality of sound emanating objects, andprocess 5200 may further comprise using the plurality of images toidentify different types of sound emanating objects and applyingdifferent conditioning for audio signals received by from differentregions associated with different types of sound emanating objects.Similarly, process 5200 may further comprise analyzing the plurality ofimages to identify an individual that speaks and a sound emanatingobject that generates background noises, and separating sounds generatedby the individual from background noises. Causing the conditioning of atleast one audio signal may include attenuating audio signals associatedwith the sound emanating object that generates background noisesrelative to the audio signals associated with the individual. Forexample, the sound emanating object may be a television or a similardevice and attenuating audio signals may include muting or reducing thevolume of audio signals associated with the television.

In some embodiments, processor 4703 may be configured to selectivelycondition audio signals associated with a plurality of individuals inthe environment of user 100. As discussed above, processor 4703 may beconfigured to apply different conditioning for different individualsbased on the information in database 5020. For example, the at least onesound emanating object may include a plurality of individuals, and theat least one processor may further be programmed to cause a firstselective conditioning of audio signals associated with a firstindividual based on retrieved information associated with the firstindividual and cause a second selective conditioning different from thefirst selective conditioning of audio signals associated with a secondindividual based on retrieved information associated with the secondindividual.

Processor 4703 may further selectively condition audio signals based onactions of the individuals. For example, the at least one soundemanating object may include a plurality of individuals and the at leastone processor may further be programmed to identify in the plurality ofimages a first individual talking to the user and a second individualtalking to a third individual. The at least one processor may amplifyaudio signals from the first individual and attenuate audio signals fromthe second individual. Accordingly, audio associated with the firstindividual, who is talking to the user, may be more easily perceptiblethan audio associated with the second individual. As another example,the at least one sound emanating object may include a plurality ofindividuals and the at least one processor may further be programmed toidentify in the plurality of images a group of individuals listening toa specific individual and to amplify audio signals from the specificindividual.

In some embodiments, processor 4703 may selectively condition audiosignals based on a detected speaker. Processor 4703 may automaticallyswitch between speakers based on another individual beginning to speak.For example, the plurality of sound emanating objects may include aplurality of individuals, and process 5200 may comprise using theplurality of images to determine that a first individual is talking;amplifying audio signals received from a region associated with thefirst individual; using the plurality of images to determine that asecond individual is about to talk and amplify audio signals receivedfrom a region associated with the second individual instead of audiosignals received from the region associated with the first individual.For example, processor 4703 may be configured to detect facial featuresof the second individual and may automatically switch to selectivelycondition audio signals associated with the second individual when theyopen their mouth, etc.

In some embodiments, processor 4703 may also determine and/or retrievevoiceprint data associated with sound emanating objects for the purposesof selectively conditioning audio associated with the sound emanatingobjects. For example, the retrieved information (e.g., informationretrieved from database 5020 in step 5240) may include a referencevoiceprint associated with the at least one sound emanating object. Insome embodiments, process 5200 may further comprise using the pluralityof images and the reference voiceprint to identify the at least onesound emanating object, separate the audio signal associated with thereference voiceprint, and cause the conditioning of the audio signalbased on predefined settings associated with an identity of the at leastone sound emanating object. For example, processor 4703 may amplify anaudio signal associated with a close family member of user 100 but mayattenuate or mute audio associated with other individuals, such as anoisy office mate. Database 5020 may store more than one voiceprint foreach sound emanating object. For example, the at least one soundemanating object may include a plurality of individuals and theretrieved information may include a set of reference voiceprints foreach individual. A first voiceprint of a specific individual may beassociated with a scenario where the specific individual is standingnext to the user, and a second voiceprint of the specific individual maybe associated with a scenario where the specific individual is talkingthrough a communication device. Accordingly, processor 4703 mayselectively condition the voice of an individual regardless of whetherthey are standing next to the user or if they are talking on a speakerphone.

The voiceprint data may also be used to improve selective conditioningof audio signals. For example, process 5200 may further compriseanalyzing the plurality of images to identify a plurality of soundemanating objects in the environment of the user, wherein each soundemanating object is associated with a unique voiceprint. Process 5200may include using voiceprints of the plurality of sound emanatingobjects to separate sounds generated by a first sound emanating objectand sounds generated by a second sound emanating object, and causing theconditioning of at least one audio signal may include attenuating audiosignals associated with the second sound emanating object relative tothe audio signals associated with the first sound emanating object.

As described above, selective conditioning may further be based oncontextual information associated with user 100 or the at least onesound emanating object. For example, process 5200 may further compriseidentifying, based on analysis of the plurality of images, a contextualsituation associated with one or more of the plurality of images;retrieving, from the database, information associated with thecontextual situation; and causing a first selective conditioning ofaudio signals from a specific object in response to a first detectedcontextual situation and cause a second selective conditioning,different from the first selective conditioning of audio signals fromthe specific object, in response to a second detected contextualsituation.

In step 5260, process 5200 may comprise causing transmission of the atleast one conditioned audio signal to a hearing interface deviceconfigured to provide sounds to an ear of the user. The conditionedaudio signal, for example, may be transmitted to hearing interfacedevice 1710, which may provide sound corresponding to the audio signalto user 100. Processor 4703 may be configured to transmit theconditioned audio signal in real time (or after a very short delay). Forexample, the at least one processor may be programmed to causetransmission of the at least one conditioned audio signal to the hearinginterface device in less than 100 mSec (e.g., 10 mSec, 20 mSec, 30 mSec,50 mSec, etc.) after the at least one audio signal was acquired by thewearable microphone. The processor performing process 1900 may furtherbe configured to cause transmission to the hearing interface device ofone or more audio signals representative of other sound emanatingobjects, which may also be conditioned. Accordingly, the hearinginterface device may comprise a receiver configured to receive at leastone audio signal. As discussed above, the at least one audio signal mayhave been acquired by a wearable microphone and may have beenselectively conditioned by at least one processor configured to receivea plurality of images captured by a wearable camera, identify at leastone sound emanating object in the plurality of images, and cause theconditioning based on retrieved information about the at least one soundemanating object. The hearing interface device may further comprise anelectroacoustic transducer configured to provide sounds from the atleast one audio signal to an ear of the user. The hearing aid device mayalso comprise other elements, such as those described above with respectto hearing interface device 1710. In some embodiments, the hearinginterface device may include a bone conduction microphone, configured toprovide an audio signal to user through vibrations of a bone of theuser's head. Such devices may be placed in contact with the exterior ofthe user's skin or may be implanted surgically and attached to the boneof the user.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, Ultra HD Blu-ray, orother optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A hearing aid system for selectively amplifyingaudio signals based on tracked lip movements, the hearing aid systemcomprising: a wearable camera configured to capture a plurality ofimages from an environment of a user; at least one microphone configuredto capture sounds from an environment of the user; and at least oneprocessor programmed to: receive the plurality of images captured by thecamera; identify a representation of at least one individual in at leastone of the plurality of images; identify at least one lip movementassociated with a mouth of the individual, based on analysis of theplurality of images; receive audio signals representative of the soundscaptured by the at least one microphone; identify, based on analysis ofthe sounds captured by the at least one microphone, at least a firstaudio signal associated with a first voice and at least a second audiosignal associated with a second voice different from the first voice;cause selective conditioning of the first audio signal based on adetermination by the at least one processor that the first audio signalis associated with the identified at least one lip movement associatedwith the mouth of the individual; and cause transmission of theselectively conditioned first audio signal to a hearing interface deviceconfigured to provide sound to an ear of the user.
 2. The system ofclaim 1, wherein the wearable camera and the at least one microphone areincluded in a common housing.
 3. The system of claim 2, wherein the atleast one processor is included in the common housing.
 4. The system ofclaim 2, wherein the at least one processor is included in a secondhousing separate from the common housing.
 5. The system of claim 4,wherein the at least one processor is configured to receive the capturedimages via a wireless link between a transmitter in the common housingand receiver in the second housing.
 6. The system of claim 1, whereinthe at least one microphone includes a directional microphone.
 7. Thesystem of claim 1, wherein the at least one microphone includes amicrophone array.
 8. The system of claim 1, wherein the hearinginterface device includes a speaker associated with an earpiece.
 9. Thesystem of claim 1, wherein the hearing interface device includes a boneconduction microphone.
 10. The system of claim 1, wherein the at leastone processor is further programmed to selectively attenuate the secondaudio signal based on a determination by the at least one processor thatthe second audio signal is not associated with the identified at leastone lip movement associated with the mouth of the individual.
 11. Thesystem of claim 1, wherein the conditioning includes attenuating one ormore audio signals received by the at least one microphone.
 12. Thesystem of claim 11, wherein the attenuated one or more audio signalsinclude the second audio signal.
 13. The system of claim 1, wherein theconditioning includes amplification of the first audio signal.
 14. Thesystem of claim 1, wherein the conditioning includes changing a toneassociated with the first audio signal.
 15. The system of claim 1,wherein the conditioning includes changing a rate of speech associatedwith the first audio signal.
 16. The system of claim 1, wherein: the atleast one individual comprises a first individual and a secondindividual; and the at least one processor is further programmed totransition from causing selective conditioning of audio signalsassociated with the first individual to causing selective conditioningof audio signals associated with the second individual based on anindication from the identified lip movement that the first individualstarted but has not finished a sentence when the second individual hasstarted speaking.
 17. A method for selectively amplifying audio signalsbased on tracked lip movements, the method comprising: receiving aplurality of images captured by a wearable camera from an environment ofthe user; identifying a representation of at least one individual in atleast one of the plurality of images; identifying at least one lipmovement associated with a mouth of the individual, based on analysis ofthe plurality of images; receiving audio signals representative of thesounds captured by at least one microphone from the environment of theuser; identifying, based on analysis of the sounds captured by the atleast one microphone, at least a first audio signal associated with afirst voice and at least a second audio signal associated with a secondvoice different from the first voice; causing selective conditioning ofthe first audio signal based on a determination that the first audiosignal is associated with the identified at least one lip movementassociated with the mouth of the individual; and causing transmission ofthe selectively conditioned first audio signal to a hearing interfacedevice configured to provide sound to an ear of the user.
 18. The methodof claim 17, wherein the wearable camera and the at least one microphoneare included in a common housing.
 19. The method of claim 17, whereinthe at least one microphone includes a directional microphone.
 20. Themethod of claim 17, wherein the at least one microphone includes amicrophone array.
 21. The method of claim 17, wherein the hearinginterface device includes a speaker associated with an earpiece.
 22. Themethod of claim 17, wherein the hearing interface device includes a boneconduction microphone.
 23. The method of claim 17, further comprisingselectively attenuating the second audio signal based on a determinationthat the second audio signal is not associated with the identified atleast one lip movement associated with the mouth of the individual. 24.The method of claim 17, wherein the conditioning includes attenuatingone or more audio signals received by the at least one microphone. 25.The method of claim 24, wherein the attenuated one or more audio signalsinclude the second audio signal.
 26. The method of claim 17, wherein theconditioning includes amplification of the first audio signal.
 27. Themethod of claim 17, wherein the conditioning includes changing a toneassociated with the first audio signal.
 28. The method of claim 17,wherein the conditioning includes changing a rate of speech associatedwith the first audio signal.
 29. The method of claim 17, wherein: the atleast one individual comprises a first individual and a secondindividual; and the method further comprises transitioning from causingselective conditioning of audio signals associated with the firstindividual to causing selective conditioning of audio signals associatedwith the second individual based on an indication from the identifiedlip movement that the first individual started but has not finished asentence when the second individual has started speaking.